tooling/hcpctl: teach the AI how to analyze node boot diagnostics by stevekuznetsov · Pull Request #6024 · Azure/ARO-HCP

stevekuznetsov · 2026-07-10T22:27:28Z

When we see ignition failures, we need to have the AI look at that data source to figure out what's gone wrong.

openshift-ci · 2026-07-10T22:27:51Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: stevekuznetsov

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

~~tooling/hcpctl/OWNERS~~ [stevekuznetsov]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

Copilot

Pull request overview

Adds node boot diagnostics (VM serial console logs) to hcpctl snapshot gathering and analysis, so the agent can cite and reason over ignition/boot failures using a first-class data source in the snapshot.

Changes:

Download *-console.log node console artifacts from the Prow job’s GCS artifacts, clean them up for readability, and write them into the snapshot under node_boot_logs/ with manifest entries and source URLs.
Extend the agent schema/validation/hydration/rendering to support log proofs sourced from node_console_log (including line excerpt extraction and artifact download links).
Add reference material and a full exemplar analysis to guide the agent on when/how to use node console logs.

Reviewed changes

Copilot reviewed 16 out of 16 changed files in this pull request and generated 4 comments.

Show a summary per file

File	Description
tooling/hcpctl/pkg/snapshot/prow.go	Adds GCS discovery/download for `*-console.log` artifacts and constructs gcsweb download URLs.
tooling/hcpctl/pkg/snapshot/manifest.go	Extends snapshot manifest schema to include `node_console_logs` and documents `node_boot_logs/` layout.
tooling/hcpctl/pkg/snapshot/gather_enriched.go	Writes node console logs into the snapshot and records them into `manifest.json`.
tooling/hcpctl/pkg/snapshot/console.go	Adds console log cleaning (ANSI stripping, GRUB preamble stripping, blank-line collapsing).
tooling/hcpctl/pkg/snapshot/console_test.go	Unit/integration tests for console log cleaning behavior.
tooling/hcpctl/pkg/snapshot/analysis.go	Writes node console logs to `node_boot_logs/` during snapshot gathering.
tooling/hcpctl/pkg/agent/validation.go	Validates `node_console_log` proof items (file existence and line range bounds).
tooling/hcpctl/pkg/agent/schema.go	Updates proof schema documentation and adds hydrated `artifact_url` for node console proofs.
tooling/hcpctl/pkg/agent/render.go	Renders node console proofs with file/line context and optional download link.
tooling/hcpctl/pkg/agent/prompts/system.md	Documents `node_console_log` as a supported `log` proof source in the output schema notes.
tooling/hcpctl/pkg/agent/prompts/references/datasources.md	Adds guidance on using node boot diagnostics as a data source and how to cite them as proof.
tooling/hcpctl/pkg/agent/prompts/exemplars/cluster-installation-ignition-failure.md	Adds a full exemplar analysis demonstrating node console log usage.
tooling/hcpctl/pkg/agent/hydration.go	Hydrates `node_console_log` excerpts and propagates artifact URLs to hydrated proof items.
tooling/hcpctl/pkg/agent/analyze.go	Wires node console log contents/URLs into validation and hydration.
tooling/hcpctl/cmd/snapshot/from_prow_job_cmd.go	Fetches node console logs from GCS during snapshot gathering for failed tests.
tooling/hcpctl/cmd/snapshot/analyze_cmd.go	Loads `node_boot_logs/` content + URLs from `manifest.json` for analysis runs.

+//  3. Collapses runs of three or more consecutive blank lines into a single
+//     blank line.


+// collapseBlankLines reduces runs of three or more consecutive blank lines
+// (lines that are empty or contain only whitespace) to a single blank line.


+		content := readFileOrEmpty(filepath.Join(o.dataDir, cl.File))
+		if content != "" {
+			nodeConsoleLogs[fileName] = content
+			nodeConsoleLogURLs[fileName] = cl.ArtifactURL
+		}


Copilot

Pull request overview

Copilot reviewed 16 out of 16 changed files in this pull request and generated 6 comments.

Copilot

Pull request overview

Copilot reviewed 16 out of 16 changed files in this pull request and generated 2 comments.

When we see ignition failures, we need to have the AI look at that data source to figure out what's gone wrong. Signed-off-by: Steve Kuznetsov <stekuznetsov@microsoft.com>

Copilot

Pull request overview

Copilot reviewed 16 out of 16 changed files in this pull request and generated 4 comments.

+		rel, err := filepath.Rel(nodeBootLogsDir, resolved)
+		if err != nil || strings.HasPrefix(rel, "..") {
+			logger.Error(fmt.Errorf("path %q resolves outside node_boot_logs directory", cl.File), "Skipping console log with suspicious path")
+			continue
+		}


+						if proof.ArtifactURL != "" {
+							sb.WriteString(fmt.Sprintf("Node console log `%s`, lines %d\u2013%d ([download](%s)):\n\n", proof.File, proof.Lines[0], proof.Lines[1], proof.ArtifactURL))
+						} else {
+							sb.WriteString(fmt.Sprintf("Node console log `%s`, lines %d\u2013%d:\n\n", proof.File, proof.Lines[0], proof.Lines[1]))
+						}


 		"conditions":     "<phase>/resources/<type>/<name>/conditions/ — status condition transition summaries (HyperShift conditions, controller conditions)",
 		"logs":           "<phase>/resources/<type>/<name>/logs/ — filtered or aggregated container and audit logs (operator logs, Maestro server/agent logs)",
 		"requests":       "<phase>/resources/<type>/<name>/requests/<METHOD>-<client_request_id>/ — per-request trace data with state/ and logs/ subdirectories",
+		"node_boot_logs": "node_boot_logs/ — VM serial console output (boot diagnostics) from nodes in the test. Files are named <node-name>-console.log. Each entry in manifest.json node_console_logs includes an artifact_url for downloading the original from the Prow job artifacts.",


+When a test creates VMs (nodes) that fail to boot or join the cluster properly,
+the test framework captures VM serial console output as `-console.log` files.
+These are available in the snapshot under `node_boot_logs/` and listed in
+`manifest.json` under `node_console_logs`.


stevekuznetsov · 2026-07-11T03:19:56Z

/retest

stevekuznetsov · 2026-07-11T13:44:38Z

/retest

stevekuznetsov · 2026-07-11T14:43:18Z

/retest

stevekuznetsov · 2026-07-11T14:46:16Z

/test all

openshift-ci · 2026-07-11T14:47:19Z

@stevekuznetsov: The following tests failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Required	Rerun command
ci/prow/image-updater-images	`827a6bd`	link	true	`/test image-updater-images`
ci/prow/periodic-healthcheck-images	`827a6bd`	link	true	`/test periodic-healthcheck-images`
ci/prow/e2e-parallel	`827a6bd`	link	true	`/test e2e-parallel`
ci/prow/test-unit	`827a6bd`	link	true	`/test test-unit`
ci/prow/lint	`827a6bd`	link	true	`/test lint`
ci/prow/config-change-detection	`827a6bd`	link	true	`/test config-change-detection`
ci/prow/integration	`827a6bd`	link	true	`/test integration`

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

Copilot AI review requested due to automatic review settings July 10, 2026 22:27

openshift-ci Bot requested review from geoberle and janboll July 10, 2026 22:27

openshift-ci Bot added the approved label Jul 10, 2026

Copilot started reviewing on behalf of stevekuznetsov July 10, 2026 22:28 View session

Copilot AI reviewed Jul 10, 2026

View reviewed changes

Copilot AI review requested due to automatic review settings July 10, 2026 22:58

stevekuznetsov force-pushed the skuznets/snapshot-node-logs branch from c90d3bd to ee50129 Compare July 10, 2026 22:58

Copilot started reviewing on behalf of stevekuznetsov July 10, 2026 22:59 View session

Copilot AI reviewed Jul 10, 2026

View reviewed changes

Copilot AI review requested due to automatic review settings July 10, 2026 23:23

stevekuznetsov force-pushed the skuznets/snapshot-node-logs branch from ee50129 to d599386 Compare July 10, 2026 23:23

Copilot started reviewing on behalf of stevekuznetsov July 10, 2026 23:24 View session

Copilot AI reviewed Jul 10, 2026

View reviewed changes

Comment thread tooling/hcpctl/cmd/snapshot/analyze_cmd.go

Comment thread tooling/hcpctl/pkg/snapshot/prow.go

tooling/hcpctl: teach the AI how to analyze node boot diagnostics

827a6bd

When we see ignition failures, we need to have the AI look at that data source to figure out what's gone wrong. Signed-off-by: Steve Kuznetsov <stekuznetsov@microsoft.com>

Copilot AI review requested due to automatic review settings July 11, 2026 00:19

stevekuznetsov force-pushed the skuznets/snapshot-node-logs branch from d599386 to 827a6bd Compare July 11, 2026 00:19

Copilot started reviewing on behalf of stevekuznetsov July 11, 2026 00:20 View session

Copilot AI reviewed Jul 11, 2026

View reviewed changes

		// 3. Collapses runs of three or more consecutive blank lines into a single
		// blank line.

		// collapseBlankLines reduces runs of three or more consecutive blank lines
		// (lines that are empty or contain only whitespace) to a single blank line.

Uh oh!

Conversation

stevekuznetsov commented Jul 10, 2026

Uh oh!

openshift-ci Bot commented Jul 10, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

stevekuznetsov commented Jul 11, 2026

Uh oh!

stevekuznetsov commented Jul 11, 2026

Uh oh!

stevekuznetsov commented Jul 11, 2026

Uh oh!

stevekuznetsov commented Jul 11, 2026

Uh oh!

openshift-ci Bot commented Jul 11, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants