fix(black-box): capture diagnostics on cancel, pin podman-compose

[gabriel-farache]

Summary

Black-box jobs (e.g. control-plane subsystem) were hitting the 25m timeout during "Start services" with little useful output — see dcm-project/control-plane#37 run.

Two related problems:

1. Hang: Rootless Podman runs healthchecks via systemd user timers. On GHA that user session is often missing, so health stays starting (Log: null) and depends_on: service_healthy never completes.
2. Silence: Log collection used if: failure(). A job timeout is a cancellation, so that step was skipped.

This PR:

• Enables linger and sets XDG_RUNTIME_DIR so Podman healthchecks can run on GHA
• Collects diagnostics on failure() || cancelled() (not on success), with continue-on-error / set +e so the step cannot flip a green job
• Dumps podman ps -a, per-container health JSON, and podman logs --tail 200 (not podman-compose logs, which needs a compose file path this workflow does not have)
• Pins podman-compose==1.6.0 to avoid version drift

[qodo-code-review]

PR Summary by Qodo

Capture black-box diagnostics on timeout/cancel and pin podman-compose

🐞 Bug fix ⚙️ Configuration changes 🕐 10-20 Minutes

AI Description

• Run container diagnostics even when the job is cancelled (timeouts aren’t failure()).
• Emit grouped Podman status, health history, and compose logs to pinpoint stuck services.
• Pin podman-compose to 1.6.0 to eliminate version drift during investigation.

Diagram

graph TD
  A(["Black-box workflow"]) --> B(["Install Podman + podman-compose 1.6.0"]) --> C(["Start services"]) --> D(["Run tests"]) --> E(["Collect diagnostics (always)"]) --> F(["Stop services (always)"])
  E --> G[("GitHub Actions logs")]

  subgraph Legend
    direction LR
    _wf(["Workflow step"]) ~~~ _log[("Log output")]
  end

Loading

High-Level Assessment

The following are alternative approaches to this PR:

1. Upload diagnostics as artifacts

• ➕ Persist large logs beyond the job console limit
• ➕ Easier offline analysis (attach compose logs, inspect JSON, etc.)
• ➖ Artifact upload may not run if cancellation becomes force-kill before upload completes
• ➖ More workflow complexity (paths, retention, size limits)

2. Use a post/cleanup composite action for diagnostics + teardown

• ➕ Centralizes reliability-critical cleanup/diagnostics across repos/jobs
• ➕ Reduces duplication and keeps workflow YAML smaller
• ➖ Requires packaging and versioning an internal action
• ➖ Still subject to cancellation grace period constraints

3. Add targeted timeouts and progress logging to the start step

• ➕ Fails fast with clearer point-of-hang (image pull vs healthcheck vs compose up)
• ➕ Reduces overall job timeout pressure
• ➖ May mask underlying slowness if timeouts are too aggressive
• ➖ Requires more detailed knowledge of service startup characteristics

Recommendation: The current approach (switching to always() and printing high-signal Podman/health/compose logs) is the most pragmatic fix for “silent” cancellations and should be merged. Consider artifact upload as a follow-up if console output proves too large, but the console-first approach is more likely to execute within GitHub’s cancellation grace period.

Files changed (1) +18 / -4

Bug fix (1) +18 / -4

black-box.yamlAlways collect Podman diagnostics on cancellation; pin podman-compose +18/-4

Always collect Podman diagnostics on cancellation; pin podman-compose

• Pins 'podman-compose' installation to version 1.6.0 to avoid version drift. Replaces the failure-only log collection step with an 'always()' diagnostics step that groups and prints 'podman ps -a', container health inspection output, and 'podman-compose logs' (tolerating errors during teardown/cancellation).

.github/workflows/black-box.yaml

[qodo-code-review]

Code Review by Qodo

🐞 Bugs (0) 📘 Rule violations (0) 📎 Requirement gaps (0) 🎨 UX issues (0) 🔗 Cross-repo conflicts (0) 📜 Skill insights (0)

1. Diagnostics can fail jobs ✓ Resolved 🐞 Bug ☼ Reliability

Description

The new "Collect diagnostics" step runs with if: always() but includes unguarded podman/`podman
inspect` commands; any non-zero exit will fail the step and can turn an otherwise successful test
run into a failed workflow. This is especially likely when Podman is unavailable (e.g., earlier
install failure) or when containers disappear between podman ps and podman inspect.

Code

.github/workflows/black-box.yaml[R64-66]

+          echo "::group::podman ps -a"
+          podman ps -a
+          echo "::endgroup::"

Evidence

The diagnostics step is now unconditional (if: always()), but only some commands are protected
with || true (health JSON inspect and podman-compose logs), while earlier podman commands are
not. This makes the step itself able to fail the job on non-zero exits.

.github/workflows/black-box.yaml[61-78]

Agent prompt

The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

### Issue description
The workflow step `Collect diagnostics` runs on `always()` and currently executes several `podman` commands without tolerating failures. In GitHub Actions, any non-zero exit code in a `run:` script fails the step, which can cause the overall job to fail even if tests passed.

### Issue Context
This step is intended to gather best-effort diagnostics (including on cancellation/timeouts), so it should never be able to flip the job result by failing itself.

### Fix Focus Areas
- .github/workflows/black-box.yaml[61-78]

### Suggested fix
- Add `|| true` (or equivalent error handling) to *all* diagnostic-only commands that should not fail the workflow:
 - `podman ps -a || true`
 - guard the loop setup: `for c in $(podman ps -aq 2>/dev/null || true); do ...`
 - guard name inspection: `name=$(podman inspect --format '{{.Name}}' "$c" 2>/dev/null || echo "$c")`
- Optionally wrap the whole diagnostics block with `set +e` / `set -e` to ensure best-effort collection without failing the step.

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

---

2. Diagnostics run on success ✓ Resolved 🐞 Bug ➹ Performance

Description

Changing the condition from failure() to always() makes the diagnostics collection execute on
successful runs too, adding log volume and runtime overhead to every blackbox job. This makes
routine green runs noisier and increases CI resource usage without improving pass/fail signal.

Code

.github/workflows/black-box.yaml[R61-63]

+      - name: Collect diagnostics
+        if: always()
        run: |
-          podman-compose logs

Evidence

The workflow explicitly sets if: always() on the diagnostics step, which causes it to run
regardless of job status (including successful runs).

.github/workflows/black-box.yaml[58-78]

Agent prompt

The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

### Issue description
The diagnostics step is currently configured with `if: always()`, so it runs even when tests succeed. This increases log volume and execution time on every successful run.

### Issue Context
The PR goal is to capture output on job timeout cancellation (which is not `failure()`). GitHub Actions supports `cancelled()` in expressions, which can be combined with `failure()` to keep the step from running on success.

### Fix Focus Areas
- .github/workflows/black-box.yaml[61-63]

### Suggested fix
- Replace `if: always()` with a condition that still triggers on cancellations/timeouts but skips success, e.g.:
 - `if: ${{ failure() || cancelled() }}`
- If you still want an override, consider adding a workflow input like `collect-diagnostics: boolean` and use:
 - `if: ${{ failure() || cancelled() || inputs.collect-diagnostics }}`

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

---

Tip of the day

💡 Did you know, you can type 'qodo, fix this' on a finding and the fix lands right on your PR

More tips ↗ | Customize Qodo ↗ | Qodo docs ↗

[gciavarrini]

@qodo review

[qodo-code-review]

review

The PR looks good from the current review state; there are no active findings.

• Diagnostics are now limited to failed or cancelled test runs via if: failure() || cancelled() (resolves finding 2).
• The diagnostics step is non-blocking with continue-on-error: true and set +e, so collection failures should not mask the original result (resolves finding 1).
• Pinning podman-compose to 1.6.0 improves dependency reproducibility.

Both previously reported findings are marked implemented.

[gciavarrini]

@qodo review

[qodo-code-review]

review

Reviewed the workflow changes. No remaining blocking issues found.

• Diagnostics are now limited to failed or cancelled jobs via if: failure() || cancelled().
• continue-on-error: true plus set +e prevents diagnostic failures from masking the original test result.
• The health and log collection loops tolerate containers disappearing during inspection.
• Rootless Podman setup now installs the pinned podman-compose version and configures lingering/XDG runtime state for healthchecks.

The previously reported concerns are addressed in the current diff; this looks good to merge.

[jordigilh]

sleep 1 after enable-linger is racy — user@<uid>.service starting (which is what actually creates /run/user/<uid>) isn't guaranteed to finish in 1s under GHA load. dcm-project/quadlet-deploy's resolve_rootless_vars.yml hit this same problem and settled on polling instead of sleeping:

- name: Start user systemd instance
  ansible.builtin.systemd_service:
    name: "user@{{ uid }}.service"
    state: started
- name: Wait for XDG_RUNTIME_DIR to exist
  ansible.builtin.wait_for:
    path: "/run/user/{{ uid }}"
    timeout: 30

Worth the shell equivalent here (e.g. timeout 30 bash -c 'until [ -d /run/user/$(id -u) ]; do sleep 0.5; done') instead of a flat sleep 1, so this doesn't reintroduce the hang intermittently instead of reliably.

[gciavarrini]

Good catch!
Addressed here 37d4059

[chadcrum]

LGMT once @jordigilh 's timing comment is handled

[jordigilh]

Verified — sleep 1 replaced with a proper wait_for-equivalent poll (timeout 30 + until [ -d /run/user/$(id -u) ]), matching the suggested pattern exactly. No further concerns.