diff --git a/.claude/agents/microshift-scenario-bootc-image-deps.md b/.claude/agents/microshift-scenario-bootc-image-deps.md new file mode 100644 index 0000000000..c8f254f8e8 --- /dev/null +++ b/.claude/agents/microshift-scenario-bootc-image-deps.md @@ -0,0 +1,318 @@ +--- +name: microshift-scenario-bootc-image-deps +description: Analyze a bootc scenario to compute all image dependencies and generate build commands +model: sonnet +color: green +--- + +# Goal +Analyze a MicroShift bootc scenario file to identify all image dependencies (direct and transitive) and produce a sorted list of `./test/bin/build_bootc_images.sh --template` commands needed to build all required images. + +**CRITICAL**: This agent ONLY works with bootc scenarios located in `test/scenarios-bootc/` directory. If the provided path does not contain "test/scenarios-bootc/", the agent MUST immediately exit with an error. DO NOT attempt to find, suggest, or convert to alternative bootc scenarios. + +# Audience +Software Engineer working with MicroShift bootc scenarios + +# Glossary + +- **bootc scenario**: A test scenario file that defines virtual machine configurations using bootc container images +- **image blueprint**: A file in `test/image-blueprints-bootc/` that defines how to build a bootc image. Two types exist: + - **containerfile** (`.containerfile`): A Containerfile with `FROM` instructions for building container images + - **image-bootc** (`.image-bootc`): A file containing an image reference (registry URL or `localhost/...` reference) used by bootc image builder (BIB) to create ISOs. May use Go templates. +- **image dependency**: When one image is based on another (referenced via `FROM localhost/...` in containerfiles or `localhost/...` in image-bootc files) +- **kickstart_image**: The image used for kickstart installation (extracted from `prepare_kickstart` calls) - mandatory +- **boot_image**: The image used to boot the VM (extracted from `launch_vm --boot_blueprint` calls or DEFAULT_BOOT_BLUEPRINT) - optional with fallback +- **target_ref_image**: Optional upgrade target image (extracted from TARGET_REF environment variable) +- **failing_ref_image**: Optional failing upgrade image (extracted from FAILING_REF environment variable) + +# Important Directories + +- `test/scenarios-bootc/`: Directory containing bootc scenario files +- `test/image-blueprints-bootc/`: Directory containing image blueprint files (`.containerfile` and `.image-bootc`) +- `test/bin/`: Directory containing build scripts + +# Workflow + +**⚠️ STOP AND READ THIS FIRST ⚠️** + +Before doing ANYTHING else, you MUST validate the scenario path. If the path does not contain "test/scenarios-bootc/", use this EXACT response template: + +```text +ERROR: Not a bootc scenario: +ERROR: This agent only works with bootc scenarios in test/scenarios-bootc/ directory +``` + +**CRITICAL INSTRUCTIONS FOR ERROR OUTPUT**: +- Replace `` with the actual file path +- Output these two lines ONLY +- Do NOT add any text before these lines +- Do NOT add any text after these lines +- Do NOT add explanations about what bootc is +- Do NOT analyze or read the provided file +- Do NOT search for alternatives +- Do NOT list other files +- Do NOT ask follow-up questions +- Stop immediately after outputting the two error lines + +**EXAMPLE - CORRECT OUTPUT when given `/test/scenarios/foo.sh`**: +```text +ERROR: Not a bootc scenario: /test/scenarios/foo.sh +ERROR: This agent only works with bootc scenarios in test/scenarios-bootc/ directory +``` + +**EXAMPLE - WRONG OUTPUT (Do NOT do this)**: +```text +ERROR: Not a bootc scenario: /test/scenarios/foo.sh + +The file you provided is in /test/scenarios/ instead of /test/scenarios-bootc/. +Would you like me to analyze one of these instead? +- /test/scenarios-bootc/presubmits/el98-src@upgrade-fails.sh +``` + +## 1. Validate Scenario File + +**CRITICAL - DO THIS FIRST**: Before ANY other processing, file reading, or analysis, validate that the scenario file is a bootc scenario: + +```bash +# Check if the scenario file path contains "scenarios-bootc" +if [[ "${scenario_file}" != *"test/scenarios-bootc/"* ]]; then + echo "ERROR: Not a bootc scenario: ${scenario_file}" >&2 + echo "ERROR: This agent only works with bootc scenarios in test/scenarios-bootc/ directory" >&2 + exit 1 +fi + +# Check if the scenario file exists +if [ ! -f "${scenario_file}" ]; then + echo "ERROR: Scenario file not found: ${scenario_file}" >&2 + exit 1 +fi +``` + +**MANDATORY RULES**: +1. Check the path BEFORE reading any files +2. Check the path BEFORE any analysis +3. If path does not contain "test/scenarios-bootc/", output ONLY the two-line error message (see Error Handling section) and STOP +4. **NEVER** explain why it's not a bootc scenario +5. **NEVER** search for alternative bootc scenarios +6. **NEVER** automatically convert or map non-bootc paths to bootc paths +7. **NEVER** suggest similar files +8. **NEVER** ask if the user wants help +9. **NEVER** provide any text beyond the exact two-line error message + +When validation fails, your entire response must be exactly two lines: the error messages. Nothing before, nothing after. + +## 2. Parse Scenario File + +Given a validated bootc scenario file path, extract the three types of images it uses: + +### 2.1 Kickstart Image (Mandatory) + +The **kickstart image** is used to create the initial VM installation via kickstart. It's extracted from `prepare_kickstart` function calls. + +```bash +# Extract kickstart_image from prepare_kickstart calls +# Example: prepare_kickstart host1 kickstart-bootc-offline.ks.template rhel96-bootc-source-ai-model-serving +# The last argument is the kickstart image name +kickstart_image="$(grep -E "prepare_kickstart.*kickstart.*" "${scenario_file}" | awk '{print $NF}')" + +# Validate that kickstart_image was found (mandatory) +if [ -z "${kickstart_image}" ]; then + echo "ERROR: No kickstart image found in scenario file: ${scenario_file}" >&2 + exit 1 +fi +``` + +### 2.2 Boot Image (Optional with Fallback) + +The **boot image** is specified in `launch_vm` calls with the `--boot_blueprint` option. If not specified, it falls back to the `DEFAULT_BOOT_BLUEPRINT` variable defined in the scenario file. + +```bash +# Extract boot_image from launch_vm --boot_blueprint calls +# Example: launch_vm --boot_blueprint rhel96-bootc-source-ai-model-serving +boot_image="$(grep -E "launch_vm.*boot_blueprint.*" "${scenario_file}" | awk '{print $NF}')" + +# If not found, extract DEFAULT_BOOT_BLUEPRINT from scenario.sh +if [ -z "${boot_image}" ]; then + boot_image="$(grep -E '^DEFAULT_BOOT_BLUEPRINT=' "test/bin/scenario.sh" | cut -d'=' -f2 | tr -d '"')" +fi +``` + +### 2.3 Upgrade Images (Optional with Default Fallback) + +**Upgrade images** are optional target images for upgrade scenarios. They can be identified by `TARGET_REF:` and/or `FAILING_REF:` tokens in the scenario file. A scenario may have both, one, or none of these. + +```bash +# Extract target_ref_image from TARGET_REF environment variable +# Example: TARGET_REF: "rhel96-bootc-upgraded" +target_ref_image="$(awk -F'TARGET_REF:' '{print $2}' "${scenario_file}" | tr -d '[:space:]"\\')" + +# Extract failing_ref_image from FAILING_REF environment variable +# Example: FAILING_REF: "rhel96-bootc-failing" +failing_ref_image="$(awk -F'FAILING_REF:' '{print $2}' "${scenario_file}" | tr -d '[:space:]"\\')" + +# Collect all upgrade images found +upgrade_images=() +[ -n "${target_ref_image}" ] && upgrade_images+=("${target_ref_image}") +[ -n "${failing_ref_image}" ] && upgrade_images+=("${failing_ref_image}") +``` + +### 2.4 Evaluate Shell Variables + +All extracted image names may be shell variables, so evaluate them by sourcing the scenario file: + +```bash +# Evaluate kickstart_image (may be a variable like ${RHEL96_BOOTC_SOURCE}) +kickstart_image="$(bash -c "source \"${scenario_file}\"; echo ${kickstart_image}")" + +# Evaluate boot_image +boot_image="$(bash -c "source \"${scenario_file}\"; echo ${boot_image}")" + +# Evaluate upgrade images (only if they exist) +if [ -n "${target_ref_image}" ]; then + target_ref_image="$(bash -c "source \"${scenario_file}\"; echo ${target_ref_image}")" +fi + +if [ -n "${failing_ref_image}" ]; then + failing_ref_image="$(bash -c "source \"${scenario_file}\"; echo ${failing_ref_image}")" +fi + +# Collect all images found +all_images=("${kickstart_image}" "${boot_image}") +[ -n "${target_ref_image}" ] && all_images+=("${target_ref_image}") +[ -n "${failing_ref_image}" ] && all_images+=("${failing_ref_image}") + +echo "Found images: kickstart=${kickstart_image} boot=${boot_image} target_ref=${target_ref_image} failing_ref=${failing_ref_image}" +``` + +**CRITICAL**: Only use the prescribed extraction commands above (grep, awk, source) to find image names. If these commands produce no results or incomplete results, report an error. Do NOT attempt creative alternatives such as reading the entire file and guessing image names, searching for patterns not described in these rules, or browsing other files to infer image names. + +## 3. Find Blueprint Files + +For each image name found, locate the corresponding blueprint file. Blueprint files can be either `.containerfile` or `.image-bootc` files: + +```bash +# Find blueprint file matching the image name (matches both .containerfile and .image-bootc extensions) +blueprint_file="$(find test/image-blueprints-bootc -type f -name "${image_name}.*" -print -quit)" +``` + +**CRITICAL**: If a blueprint file is not found using the exact `find` command above, report an error immediately. Do NOT attempt any alternative search strategies such as: +- Searching with partial or fuzzy image names +- Browsing directories manually to find similar files +- Stripping prefixes/suffixes from image names to find matches +- Using `grep` to search file contents for references +- Listing directory contents to guess matching files +- Any other creative approach to locate the blueprint + +## 4. Find Dependencies Recursively + +For each blueprint file, recursively find all dependencies. Both `.containerfile` and `.image-bootc` files can reference `localhost/` images: +- In `.containerfile` files: `FROM localhost/image-name:latest` +- In `.image-bootc` files: `localhost/image-name:latest` (bare reference, no `FROM` prefix) + +The same grep pattern works for both file types: + +```bash +# Extract localhost dependencies from the blueprint file +# Works for both .containerfile (FROM localhost/...) and .image-bootc (localhost/...) files +deps="$(grep -Eo 'localhost/[a-zA-Z0-9-]+:latest' "${blueprint_file}" | \ + awk -F'localhost/' '{print $2}' | sed 's/:latest//')" + +# For each dependency, find its blueprint file and recurse +for dep in ${deps}; do + dep_file="$(find test/image-blueprints-bootc -type f -name "${dep}.*" -print -quit)" + # Recursively process dep_file to find its dependencies +done +``` + +**Important**: Track all processed files to avoid infinite loops and duplicates. + +**Note on `.image-bootc` dependencies**: An `.image-bootc` file may reference a `localhost/` image that is built from a `.containerfile`. For example, `rhel98-bootc.image-bootc` contains `localhost/rhel98-test-agent:latest`, which depends on the `rhel98-test-agent.containerfile` blueprint. Follow these cross-type dependencies the same way. + +## 5. Generate Build Commands + +For each unique blueprint file (dependencies first, then the main images), generate: + +```bash +# For .containerfile blueprints: +./test/bin/build_bootc_images.sh --template /path/to/blueprint.containerfile +# For .image-bootc blueprints: +./test/bin/build_bootc_images.sh --template /path/to/blueprint.image-bootc +``` + +**Important**: +- Sort the output so dependencies are built before images that depend on them +- Use absolute paths for blueprint files (use `realpath`) +- Output commands in a deterministic, sorted order +- NEVER actually execute `./test/bin/build_bootc_images.sh` - only output the commands + +## 6. Output Format + +The final output should be a sorted list of build commands, one per line. The file extension in the path reflects the actual blueprint type (`.containerfile` or `.image-bootc`): + +```bash +./test/bin/build_bootc_images.sh --template microshift/test/image-blueprints-bootc/layer1-base/group1/rhel98-test-agent.containerfile +./test/bin/build_bootc_images.sh --template microshift/test/image-blueprints-bootc/layer1-base/group2/rhel98-bootc.image-bootc +./test/bin/build_bootc_images.sh --template microshift/test/image-blueprints-bootc/layer2-presubmit/group1/rhel98-bootc-source.containerfile +``` + +# Tips + +1. **CRITICAL**: Validate that the scenario file path contains `scenarios-bootc` BEFORE any processing. Exit with error if not. +2. **DO NOT** attempt to find or convert non-bootc scenarios to bootc scenarios. Report error immediately. +3. Use `grep -Eo 'localhost/[a-zA-Z0-9-]+:latest'` to extract localhost image references from both `.containerfile` and `.image-bootc` files +4. Use `realpath` to convert relative paths to absolute paths +5. Use `sort -u` to ensure unique, sorted output +6. Maintain a set of processed blueprint files to avoid duplicates and infinite recursion +7. Dependencies must appear before images that depend on them in the output +8. If a blueprint file is not found, report an error with the image name. Do NOT attempt alternative search strategies to find it +9. Blueprint files can be `.containerfile` or `.image-bootc` - the `find` with `"${image_name}.*"` matches both +10. An `.image-bootc` file's `localhost/` dependency may resolve to a `.containerfile` blueprint (cross-type dependency) + +# Error Handling + +**CRITICAL VALIDATION**: The first check must be whether the scenario is a bootc scenario. Report errors in this order: + +1. If the scenario is not a bootc scenario (path does not contain `scenarios-bootc`): + + **YOUR ENTIRE RESPONSE MUST BE EXACTLY**: + ```text + ERROR: Not a bootc scenario: ${scenario_file} + ERROR: This agent only works with bootc scenarios in test/scenarios-bootc/ directory + ``` + + **NOTHING ELSE**. Your response must contain ONLY these two error lines. **FORBIDDEN**: + - Explanations about the error + - Descriptions of what bootc scenarios are + - Search for similar bootc scenarios + - Suggestions for alternative files + - Recommendations or help + - Lists of available bootc scenarios + - Questions to the user + - Any additional text whatsoever + + Just the two error lines above, then STOP. + +2. If the scenario file doesn't exist, report: "ERROR: Scenario file not found: ${scenario_file}" + +3. If no images are found in the scenario, report: "ERROR: No dependencies found for scenario file: ${scenario_file}" + +4. If a blueprint file is not found for an image, report: "ERROR: Image file not found: ${image_name}" + +# Example Usage + +Input: `microshift/test/scenarios-bootc/periodics/el96-src@ai-model-serving-offline.sh` + +Expected workflow: +1. Parse scenario → finds kickstart image `rhel96-bootc-source-ai-model-serving` and boot image `rhel96-bootc-source-ai-model-serving` +2. Find blueprints → locates `rhel96-bootc-source-ai-model-serving.containerfile` and `rhel96-bootc-source-ai-model-serving.image-bootc` +3. Check `.image-bootc` dependencies → finds `localhost/rhel96-bootc-source-ai-model-serving:latest` +4. Check `.containerfile` dependencies → finds `FROM localhost/rhel96-bootc-source:latest` +5. Recurse → finds `rhel96-bootc-source.containerfile` which depends on `localhost/rhel96-test-agent:latest` +6. Recurse → finds `rhel96-test-agent.containerfile` which depends on an external registry image (no further localhost dependencies) +7. Output sorted commands (dependencies first): + ```bash + ./test/bin/build_bootc_images.sh --template .../layer1-base/group1/rhel96-test-agent.containerfile + ./test/bin/build_bootc_images.sh --template .../layer2-presubmit/group1/rhel96-bootc-source.containerfile + ./test/bin/build_bootc_images.sh --template .../layer3-periodic/group1/rhel96-bootc-source-ai-model-serving.containerfile + ./test/bin/build_bootc_images.sh --template .../layer3-periodic/group2/rhel96-bootc-source-ai-model-serving.image-bootc + ``` diff --git a/.claude/agents/microshift-scenario-ostree-image-deps.md b/.claude/agents/microshift-scenario-ostree-image-deps.md new file mode 100644 index 0000000000..4d81b95533 --- /dev/null +++ b/.claude/agents/microshift-scenario-ostree-image-deps.md @@ -0,0 +1,380 @@ +--- +name: microshift-scenario-ostree-image-deps +description: Analyze an ostree scenario to compute all image dependencies and generate build commands +model: sonnet +color: blue +--- + +# Goal +Analyze a MicroShift ostree scenario file to identify all image dependencies (direct and transitive) and produce a sorted list of `./test/bin/build_images.sh -t` commands needed to build all required images. + +**CRITICAL**: This agent ONLY works with ostree scenarios located in `test/scenarios/` directory. If the provided path does not contain "test/scenarios/", the agent MUST immediately exit with an error. DO NOT attempt to find, suggest, or convert to alternative ostree scenarios. + +# Audience +Software Engineer working with MicroShift ostree scenarios + +# Glossary + +- **ostree scenario**: A test scenario file that defines virtual machine configurations using ostree commit images +- **image blueprint**: A TOML file that defines how to build an ostree image using osbuild-composer +- **image-installer file** (`.image-installer`): A file that contains the image name to be built as an ISO installer. Shares the same basename as its corresponding `.toml` blueprint and is processed automatically alongside it +- **alias file** (`.alias`): A file that defines an image name alias. The filename (without extension) is the alias image name, and the file contents is the real image name it resolves to. For example, `rhel-9.6-microshift-source-aux.alias` contains `rhel-9.6-microshift-source`, meaning image `rhel-9.6-microshift-source-aux` is an alias for `rhel-9.6-microshift-source` +- **image dependency**: When one image is based on another (referenced via `# parent = "..."`). The `# parent` directive is a pseudo-directive that is always commented out — it is NOT a TOML key. It is parsed directly from the comment by `build_images.sh`. A commented `# parent = "..."` line is ALWAYS an active dependency and must never be ignored +- **kickstart_image**: The image used for kickstart installation (extracted from `prepare_kickstart` calls) - mandatory +- **boot_image**: The image used to boot the VM (extracted from `launch_vm --boot_blueprint` calls) - optional +- **target_ref_image**: Optional upgrade target image (extracted from `--variable "TARGET_REF:..."` in run_tests calls) +- **failing_ref_image**: Optional failing upgrade image (extracted from `--variable "FAILING_REF:..."` in run_tests calls) + +# Important Directories + +- `test/scenarios/`: Directory containing ostree scenario files +- `test/image-blueprints/`: Directory containing image blueprint TOML files +- `test/bin/`: Directory containing build scripts + +# Workflow + +**⚠️ STOP AND READ THIS FIRST ⚠️** + +Before doing ANYTHING else, you MUST validate the scenario path. If the path does not contain "test/scenarios/", use this EXACT response template: + +```text +ERROR: Not an ostree scenario: +ERROR: This agent only works with ostree scenarios in test/scenarios/ directory +``` + +**CRITICAL INSTRUCTIONS FOR ERROR OUTPUT**: +- Replace `` with the actual file path +- Output these two lines ONLY +- Do NOT add any text before these lines +- Do NOT add any text after these lines +- Do NOT add explanations about what ostree is +- Do NOT analyze or read the provided file +- Do NOT search for alternatives +- Do NOT list other files +- Do NOT ask follow-up questions +- Stop immediately after outputting the two error lines + +**EXAMPLE - CORRECT OUTPUT when given `/test/scenarios-bootc/foo.sh`**: +```text +ERROR: Not an ostree scenario: /test/scenarios-bootc/foo.sh +ERROR: This agent only works with ostree scenarios in test/scenarios/ directory +``` + +**EXAMPLE - WRONG OUTPUT (Do NOT do this)**: +```text +ERROR: Not an ostree scenario: /test/scenarios-bootc/foo.sh + +The file you provided is in /test/scenarios-bootc/ instead of /test/scenarios/. +Would you like me to analyze one of these instead? +- /test/scenarios/presubmits/el96-src@upgrade-ok.sh +``` + +## 1. Validate Scenario File + +**CRITICAL - DO THIS FIRST**: Before ANY other processing, file reading, or analysis, validate that the scenario file is an ostree scenario: + +```bash +# Check if the scenario file path contains "test/scenarios/" and does NOT contain "test/scenarios-bootc" +if [[ "${scenario_file}" != *"test/scenarios/"* ]] ; then + echo "ERROR: Not an ostree scenario: ${scenario_file}" >&2 + echo "ERROR: This agent only works with ostree scenarios in test/scenarios/ directory" >&2 + exit 1 +fi + +# Check if the scenario file exists +if [ ! -f "${scenario_file}" ]; then + echo "ERROR: Scenario file not found: ${scenario_file}" >&2 + exit 1 +fi +``` + +**MANDATORY RULES**: +1. Check the path BEFORE reading any files +2. Check the path BEFORE any analysis +3. If path does not contain "test/scenarios/", output ONLY the two-line error message (see Error Handling section) and STOP +4. **NEVER** explain why it's not an ostree scenario +5. **NEVER** search for alternative ostree scenarios +6. **NEVER** automatically convert or map non-ostree paths to ostree paths +7. **NEVER** suggest similar files +8. **NEVER** ask if the user wants help +9. **NEVER** provide any text beyond the exact two-line error message + +When validation fails, your entire response must be exactly two lines: the error messages. Nothing before, nothing after. + +## 2. Parse Scenario File + +Given a validated ostree scenario file path, extract the types of images it uses: + +### 2.1 Kickstart Image (Mandatory) + +The **kickstart image** is used to create the initial VM installation via kickstart. It's extracted from `prepare_kickstart` function calls. + +```bash +# Extract kickstart_image from prepare_kickstart calls +# Example: prepare_kickstart host1 kickstart.ks.template rhel-9.6-microshift-source +# The last argument is the kickstart image name +kickstart_image="$(grep -E "prepare_kickstart.*kickstart.*" "${scenario_file}" | awk '{print $NF}')" + +# Validate that kickstart_image was found (mandatory) +if [ -z "${kickstart_image}" ]; then + echo "ERROR: No kickstart image found in scenario file: ${scenario_file}" >&2 + exit 1 +fi +``` + +### 2.2 Boot Image (Optional with Default Fallback) + +The **boot image** is specified in `launch_vm` calls with the `--boot_blueprint` option. This is optional for ostree scenarios. + +```bash +# Extract boot_image from launch_vm --boot_blueprint calls +# Example: launch_vm --boot_blueprint rhel-9.6-microshift-source-isolated +boot_image="$(grep -E "launch_vm.*boot_blueprint.*" "${scenario_file}" | awk '{print $NF}')" + +# If not found, extract DEFAULT_BOOT_BLUEPRINT from scenario.sh +if [ -z "${boot_image}" ]; then + boot_image="$(grep -E '^DEFAULT_BOOT_BLUEPRINT=' "test/bin/scenario.sh" | cut -d'=' -f2 | tr -d '"')" +fi +``` + +### 2.3 Upgrade Images (Optional) + +**Upgrade images** are optional target images for upgrade scenarios. They are extracted from `--variable` flags in `run_tests` calls. A scenario may have both, one, or none of these. + +```bash +# Extract target_ref_image from --variable "TARGET_REF:..." in run_tests calls +# Example: --variable "TARGET_REF:rhel-9.6-microshift-source" +target_ref_image="$(awk -F'TARGET_REF:' '{print $2}' "${scenario_file}" | tr -d '[:space:]"\\')" + +# Extract failing_ref_image from --variable "FAILING_REF:..." in run_tests calls +# Example: --variable "FAILING_REF:rhel-9.6-microshift-source" +failing_ref_image="$(awk -F'FAILING_REF:' '{print $2}' "${scenario_file}" | tr -d '[:space:]"\\')" + +# Collect all upgrade images found +upgrade_images=() +[ -n "${target_ref_image}" ] && upgrade_images+=("${target_ref_image}") +[ -n "${failing_ref_image}" ] && upgrade_images+=("${failing_ref_image}") +``` + +### 2.4 Evaluate Shell Variables + +All extracted image names may be shell variables, so evaluate them by sourcing the scenario file: + +```bash +# Evaluate kickstart_image (may be a variable or expression) +kickstart_image="$(bash -c "source \"${scenario_file}\"; echo ${kickstart_image}")" + +# Evaluate boot_image +boot_image="$(bash -c "source \"${scenario_file}\"; echo ${boot_image}")" + +# Evaluate upgrade images (only if they exist) +if [ -n "${target_ref_image}" ]; then + target_ref_image="$(bash -c "source \"${scenario_file}\"; echo ${target_ref_image}")" +fi + +if [ -n "${failing_ref_image}" ]; then + failing_ref_image="$(bash -c "source \"${scenario_file}\"; echo ${failing_ref_image}")" +fi + +# Collect all images found +all_images=("${kickstart_image}") +[ -n "${boot_image}" ] && all_images+=("${boot_image}") +[ -n "${target_ref_image}" ] && all_images+=("${target_ref_image}") +[ -n "${failing_ref_image}" ] && all_images+=("${failing_ref_image}") + +echo "Found images: kickstart=${kickstart_image} boot=${boot_image} target_ref=${target_ref_image} failing_ref=${failing_ref_image}" +``` + +**CRITICAL**: Only use the prescribed extraction commands above (grep, awk, source) to find image names. If these commands produce no results or incomplete results, report an error. Do NOT attempt creative alternatives such as reading the entire file and guessing image names, searching for patterns not described in these rules, or browsing other files to infer image names. + +## 3. Find Blueprint Files + +For each image name found, locate the corresponding blueprint file. Image names are found by searching file contents, not by matching filenames. + +There are three types of files that produce image names: +- **TOML files** (`.toml`): The image name is defined by the `name = "..."` field at the top of the file (e.g., `rhel96.toml` contains `name = "rhel-9.6"`) +- **Image-installer files** (`.image-installer`): The file contents IS the image name (e.g., `rhel96.image-installer` contains `rhel-9.6`) +- **Alias files** (`.alias`): The filename (without extension) is an alias image name, and the file contents is the real image name it resolves to (e.g., `rhel-9.6-microshift-source-aux.alias` contains `rhel-9.6-microshift-source`) + +Search TOML files first, then `.image-installer` files, then `.alias` files: + +```bash +# Step 1: Search for ^name = "" inside TOML files +blueprint_file="$(grep -rl "^name = \"${image_name}\"" test/image-blueprints/ --include="*.toml" | head -1)" + +# Step 2: If not found in TOML files, search .image-installer files for the image name +# The .image-installer file content is the image name itself (one name per file) +if [ -z "${blueprint_file}" ]; then + blueprint_file="$(grep -rl "^${image_name}$" test/image-blueprints/ --include="*.image-installer" | head -1)" +fi + +# Step 3: If still not found, check if the image name is an alias +# Alias files are named .alias and contain the real image name +if [ -z "${blueprint_file}" ]; then + blueprint_file="$(find test/image-blueprints -type f -name "${image_name}.alias" -print -quit)" +fi + +if [ -z "${blueprint_file}" ]; then + echo "ERROR: No blueprint file found for image: ${image_name}" >&2 + exit 1 +fi +``` + +**CRITICAL**: If a blueprint file is not found using the exact search commands above, report an error immediately. Do NOT attempt any alternative search strategies such as: +- Searching with partial or fuzzy image names +- Browsing directories manually to find similar files +- Stripping prefixes/suffixes from image names to find matches +- Matching by filename instead of file contents +- Listing directory contents to guess matching files +- Any other creative approach to locate the blueprint + +## 4. Find Dependencies Recursively + +For each blueprint file, extract parent dependencies and recursively resolve them. + +### 4.1 Extract Parent Dependency + +Run this command to extract the parent image name from a blueprint file: + +```bash +deps="$(grep -E '^[[:space:]]*#[[:space:]]*parent[[:space:]]*=' "${blueprint_file}" | \ + sed -n 's/.*parent[[:space:]]*=[[:space:]]*"\([^"]*\)".*/\1/p')" +``` + +**⚠️ MANDATORY RULE**: If the above command produces any output, that output is a **real, active parent dependency**. You MUST: +1. Treat it as a required dependency +2. Resolve it and find its blueprint file +3. Include it in the build commands + +**⚠️ FORBIDDEN**: You must NEVER: +- Describe `# parent` lines as "commented out" — the `#` is part of the directive syntax, not a comment +- Skip a parent dependency for any reason +- Say the blueprint "has no active dependencies" when the grep command found a match + +The `# parent = "..."` syntax is a pseudo-directive parsed by `build_images.sh`. The `#` prefix is required because `parent` is not a valid TOML key. Every `# parent` line is active by design. + +### 4.2 Resolve Go Template Expressions in Parent Values + +Parent values may contain Go template expressions (e.g., `{{ .Env.PREVIOUS_MINOR_VERSION }}`). These MUST be resolved before searching for the blueprint: + +1. Read `test/bin/common_versions.sh` to get version variables +2. Replace `{{ .Env.VARNAME }}` patterns with the shell variable values + +Key variables from `test/bin/common_versions.sh`: +- `MINOR_VERSION` — current minor version (e.g., `22`) +- `PREVIOUS_MINOR_VERSION` — computed as `MINOR_VERSION - 1` (e.g., `21`) +- `YMINUS2_MINOR_VERSION` — computed as `MINOR_VERSION - 2` (e.g., `20`) +- `FAKE_NEXT_MINOR_VERSION` — computed as `MINOR_VERSION + 1` (e.g., `23`) + +**Example**: +- Raw: `rhel-9.6-microshift-brew-optionals-4.{{ .Env.PREVIOUS_MINOR_VERSION }}-zstream` +- Read `common_versions.sh` → `MINOR_VERSION=22`, so `PREVIOUS_MINOR_VERSION=21` +- Resolved: `rhel-9.6-microshift-brew-optionals-4.21-zstream` + +### 4.3 Find Blueprint for Parent Dependency + +For each resolved parent dependency, find its blueprint file using the same three-step search as Section 3: + +```bash +for dep in ${deps}; do + dep_file="$(grep -rl "^name = \"${dep}\"" test/image-blueprints/ --include="*.toml" | head -1)" + if [ -z "${dep_file}" ]; then + dep_file="$(grep -rl "^${dep}$" test/image-blueprints/ --include="*.image-installer" | head -1)" + fi + if [ -z "${dep_file}" ]; then + dep_file="$(find test/image-blueprints -type f -name "${dep}.alias" -print -quit)" + fi + if [ -z "${dep_file}" ]; then + echo "ERROR: No blueprint file found for image: ${dep}" >&2 + exit 1 + fi + # Recursively process dep_file to find its dependencies (go back to step 4.1) +done +``` + +**Important**: Track all processed files to avoid infinite loops and duplicates. + +## 5. Generate Build Commands + +For each unique blueprint file (dependencies first, then the main images), generate: + +```bash +./test/bin/build_images.sh -t /path/to/blueprint.toml +``` + +**Important**: +- Sort the output so dependencies are built before images that depend on them +- Use absolute paths for blueprint files (use `realpath`) +- Output commands in a deterministic, sorted order +- NEVER actually execute `build_images.sh` - only output the commands + +## 6. Output Format + +The final output should be a sorted list of build commands, one per line: + +```bash +./test/bin/build_images.sh -t /home/microshift/microshift/test/image-blueprints/layer1-base/group1/rhel96.toml +./test/bin/build_images.sh -t /home/microshift/microshift/test/image-blueprints/layer2-presubmit/group1/rhel96-source-base.toml +./test/bin/build_images.sh -t /home/microshift/microshift/test/image-blueprints/layer3-periodic/group1/rhel96-microshift-source.toml +``` + +# Tips + +1. **CRITICAL**: Validate that the scenario file path contains `test/scenarios/` and does NOT contain `scenarios-bootc` BEFORE any processing. Exit with error if validation fails. +2. **DO NOT** attempt to find or convert non-ostree scenarios to ostree scenarios. Report error immediately. +3. Use `grep -E '^[[:space:]]*#[[:space:]]*parent[[:space:]]*='` to extract parent image references. The `#` is NOT a comment — it is part of the pseudo-directive syntax. These lines are ALWAYS active dependencies +4. Use `realpath` to convert relative paths to absolute paths +5. Use `sort -u` to ensure unique, sorted output +6. Maintain a set of processed blueprint files to avoid duplicates and infinite recursion +7. Dependencies must appear before images that depend on them in the output +8. If a blueprint file is not found, report an error with the image name. Do NOT attempt alternative search strategies to find it +9. **IMPORTANT**: Blueprint files are found by searching file contents, NOT by matching filenames. Search `^name = ""` in `.toml` files first, then `^$` in `.image-installer` files, then `.alias` by filename. The filename does not necessarily match the image name (e.g., `rhel96.toml` produces image `rhel-9.6`) +10. **`.image-installer` files**: These contain the image name directly (file content = image name). When a match is found in a `.image-installer` file, the corresponding `.toml` blueprint has the same basename (e.g., `rhel96.image-installer` → `rhel96.toml`) +11. **`.alias` files**: These define image name aliases. The filename (without extension) is the alias, and the file contents is the real image name. When an alias is found, resolve the real image name and restart the search from Step 1. For example, `rhel-9.6-microshift-source-aux.alias` contains `rhel-9.6-microshift-source`, so looking up `rhel-9.6-microshift-source-aux` resolves to the blueprint for `rhel-9.6-microshift-source` + +# Error Handling + +**CRITICAL VALIDATION**: The first check must be whether the scenario is an ostree scenario. Report errors in this order: + +1. If the scenario is not an ostree scenario (path does not contain `test/scenarios/` OR contains `scenarios-bootc`): + + **YOUR ENTIRE RESPONSE MUST BE EXACTLY**: + ```text + ERROR: Not an ostree scenario: ${scenario_file} + ERROR: This agent only works with ostree scenarios in test/scenarios/ directory + ``` + + **NOTHING ELSE**. Your response must contain ONLY these two error lines. **FORBIDDEN**: + - Explanations about the error + - Descriptions of what ostree scenarios are + - Search for similar ostree scenarios + - Suggestions for alternative files + - Recommendations or help + - Lists of available ostree scenarios + - Questions to the user + - Any additional text whatsoever + + Just the two error lines above, then STOP. + +2. If the scenario file doesn't exist, report: "ERROR: Scenario file not found: ${scenario_file}" + +3. If no images are found in the scenario, report: "ERROR: No dependencies found for scenario file: ${scenario_file}" + +4. If a blueprint file is not found for an image, report: "ERROR: Image file not found: ${image_name}" + +# Example Usage + +Input: `/home/microshift/microshift/test/scenarios/periodics/el96-src@greenboot.sh` + +Expected workflow: +1. Parse scenario → finds `rhel-9.6-microshift-source` image +2. Find blueprint → locates `rhel-9.6-microshift-source.toml` +3. Check dependencies → finds `# parent = "rhel-9.6-microshift-4.18"` +4. Recurse → finds `rhel-9.6-microshift-4.18.toml` which depends on `rhel-9.6` +5. Recurse → finds `rhel-9.6.toml` which has no dependencies +6. Output sorted commands: + ```bash + ./test/bin/build_images.sh -t .../rhel96.toml + ./test/bin/build_images.sh -t .../rhel-9.6-microshift-4.18.toml + ./test/bin/build_images.sh -t .../rhel-9.6-microshift-source.toml + ``` diff --git a/.claude/settings.json b/.claude/settings.json index e85447cce0..f6e6d3569f 100644 --- a/.claude/settings.json +++ b/.claude/settings.json @@ -4,6 +4,11 @@ "Read(//tmp/**)", "Bash(mktemp:*)", "Bash(curl:*)", + "Bash(grep:*)", + "Bash(find:*)", + "Bash(awk:*)", + "Bash(sed:*)", + "Bash(tr:*)", "Bash(gcloud storage cp:*)", "Bash(gh pr view:*)", "Bash(gh pr diff:*)", diff --git a/test/Makefile b/test/Makefile index 41b88fe85f..a63477dcc6 100644 --- a/test/Makefile +++ b/test/Makefile @@ -30,3 +30,75 @@ build-base-branch: .PHONY: robotidy robotidy: cd .. && make verify-rf + +#################################################### +# Testing Framework commands for local development # +#################################################### + +# +# Configuration commands +# +.PHONY: test-config-composer +test-config-composer: + ./bin/manage_composer_config.sh cleanup + ./bin/manage_composer_config.sh create + ./bin/manage_composer_config.sh create-workers + +.PHONY: test-config-hypervisor +test-config-hypervisor: + ./bin/manage_hypervisor_config.sh cleanup + ./bin/manage_hypervisor_config.sh create + +.PHONY: test-cache-download +test-cache-download: + bash -euo pipefail -c '\ + source ./bin/common.sh ; \ + export AWS_BUCKET_NAME=microshift-build-cache-us-west-2 ; \ + ./test/bin/manage_build_cache.sh download -b "$${SCENARIO_BUILD_BRANCH}" -t "$${SCENARIO_BUILD_TAG}" ; \ + ' + +# find_layer: finds a layer directory matching the token +# Usage: $(call find_layer,base_dir,token) +# Example: $(call find_layer,image-blueprints,base) returns ./image-blueprints/layer1-base +define find_layer +$(shell find $(1) -maxdepth 1 -type d -name '*$(2)' -print) +endef + +# +# Build commands +# +.PHONY: test-build-registry-clean +test-build-registry-clean: + ./bin/manage_composer_config.sh cleanup + rm -rf "$(REPO)/_output/test-images/mirror-registry" + ./bin/manage_webserver.sh start + +PHONY: test-build-ostree-presubmit +test-build-ostree-presubmit: + ./bin/build_images.sh -l $(call find_layer,image-blueprints,base) $(ARGS) + ./bin/build_images.sh -l $(call find_layer,image-blueprints,presubmit) $(ARGS) + +.PHONY: test-build-ostree-periodic +test-build-ostree-periodic: + ./bin/build_images.sh -l $(call find_layer,image-blueprints,periodic) $(ARGS) + +.PHONY: test-build-ostree-release +test-build-ostree-release: + ./bin/build_images.sh -l $(call find_layer,image-blueprints,release) $(ARGS) + +.PHONY: test-build-bootc-presubmit +test-build-bootc-presubmit: + ./bin/build_bootc_images.sh -l $(call find_layer,image-blueprints-bootc,base) $(ARGS) + ./bin/build_bootc_images.sh -l $(call find_layer,image-blueprints-bootc,presubmit) $(ARGS) + +.PHONY: test-build-bootc-periodic +test-build-bootc-periodic: + ./bin/build_bootc_images.sh -l $(call find_layer,image-blueprints-bootc,periodic) $(ARGS) + +.PHONY: test-build-bootc-release +test-build-bootc-release: + ./bin/build_bootc_images.sh -l $(call find_layer,image-blueprints-bootc,release) $(ARGS) + +.PHONY: test-build-bootc-upstream +test-build-bootc-upstream: + ./bin/build_bootc_images.sh -l $(call find_layer,image-blueprints-bootc,upstream) $(ARGS) diff --git a/test/bin/scenario_builder.sh b/test/bin/scenario_builder.sh new file mode 100755 index 0000000000..13c84ac66e --- /dev/null +++ b/test/bin/scenario_builder.sh @@ -0,0 +1,175 @@ +#!/bin/bash +# +# Analyze a MicroShift scenario file (bootc or ostree) to identify all image +# dependencies (direct and transitive) and produce a sorted list of build +# commands needed to build all required images. +# +# Usage: scenario_builder.sh +# +set -euo pipefail + +SCRIPTDIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )" +REPO_ROOT="$(cd "${SCRIPTDIR}/../.." && pwd)" + +# Validate arguments +if [ $# -ne 1 ]; then + echo "ERROR: Usage: $0 " >&2 + exit 1 +fi + +scenario_file="${1}" + +# ============================================================================ +# Step 1: Validate Scenario File Path and Detect Scenario Type +# ============================================================================ + +# Detect scenario type based on path +if [[ "${scenario_file}" == *"test/scenarios-bootc/"* ]]; then + scenario_type="bootc" +elif [[ "${scenario_file}" == *"test/scenarios/"* ]]; then + scenario_type="ostree" +else + echo "ERROR: Unknown scenario type: ${scenario_file}" >&2 + echo "ERROR: Scenario must be in test/scenarios-bootc/ or test/scenarios/ directory" >&2 + exit 1 +fi + +# Check if the scenario file exists +if [ ! -f "${scenario_file}" ]; then + echo "ERROR: Scenario file not found: ${scenario_file}" >&2 + exit 1 +fi + +# ============================================================================ +# Step 2: Parse Scenario File to Extract Image Names +# ============================================================================ + +# 2.1 Extract kickstart_image (mandatory) +kickstart_image="$(grep -E "prepare_kickstart.*kickstart.*" "${scenario_file}" | awk '{print $NF}' || true)" + +if [ -z "${kickstart_image}" ]; then + echo "ERROR: No kickstart image found in scenario file: ${scenario_file}" >&2 + exit 1 +fi + +# 2.2 Extract boot_image (optional with fallback) +boot_image="$(grep -E "launch_vm.*boot_blueprint.*" "${scenario_file}" | grep -oE '\-\-boot_blueprint[[:space:]]+[^[:space:]]+' | awk '{print $2}' || true)" + +# If not found, use DEFAULT_BOOT_BLUEPRINT from scenario.sh +if [ -z "${boot_image}" ]; then + boot_image="$(grep -E '^DEFAULT_BOOT_BLUEPRINT=' "${REPO_ROOT}/test/bin/scenario.sh" | cut -d'=' -f2 | tr -d '"' || true)" +fi + +# 2.3 Extract upgrade images (optional) +target_ref_image="$(awk -F'TARGET_REF:' '{print $2}' "${scenario_file}" | tr -d '\\[:space:]"' || true)" +failing_ref_image="$(awk -F'FAILING_REF:' '{print $2}' "${scenario_file}" | tr -d '\\[:space:]"' || true)" + +# 2.4 Evaluate shell variables by sourcing the scenario file +kickstart_image="$(bash -c "source \"${scenario_file}\"; echo ${kickstart_image}" 2>/dev/null || echo "${kickstart_image}")" +boot_image="$(bash -c "source \"${scenario_file}\"; echo ${boot_image}" 2>/dev/null || echo "${boot_image}")" + +if [ -n "${target_ref_image}" ]; then + target_ref_image="$(bash -c "source \"${scenario_file}\"; echo ${target_ref_image}" 2>/dev/null || echo "${target_ref_image}")" +fi + +if [ -n "${failing_ref_image}" ]; then + failing_ref_image="$(bash -c "source \"${scenario_file}\"; echo ${failing_ref_image}" 2>/dev/null || echo "${failing_ref_image}")" +fi + +# Collect all images found +all_images=("${kickstart_image}") +[ -n "${boot_image}" ] && all_images+=("${boot_image}") +[ -n "${target_ref_image}" ] && all_images+=("${target_ref_image}") +[ -n "${failing_ref_image}" ] && all_images+=("${failing_ref_image}") + +# Debug output (commented for production) +# echo "Found images: kickstart=${kickstart_image} boot=${boot_image} target_ref=${target_ref_image} failing_ref=${failing_ref_image}" >&2 + +# ============================================================================ +# Step 3 & 4: Find Blueprint Files and Dependencies Recursively +# ============================================================================ + +# Track processed blueprints to avoid infinite loops +declare -A processed_blueprints +# Track ordered list of blueprint files +declare -a blueprint_order + +# Recursive function to find dependencies +find_dependencies_bootc() { + local image_name="${1}" + + # Find all blueprint files matching the image name (may have both .containerfile and .image-bootc) + local blueprint_files=() + while IFS= read -r -d '' file; do + blueprint_files+=("${file}") + done < <(find "${REPO_ROOT}/test/image-blueprints-bootc" -type f -name "${image_name}.*" \( -name "*.containerfile" -o -name "*.image-bootc" \) -print0 2>/dev/null | sort -z) + + if [ ${#blueprint_files[@]} -eq 0 ]; then + echo "ERROR: No blueprint file found for image: ${image_name}" >&2 + exit 1 + fi + + # Process each blueprint file for this image + for blueprint_file in "${blueprint_files[@]}"; do + # Convert to absolute path + local abs_blueprint_file + abs_blueprint_file="$(realpath "${blueprint_file}")" + + # Skip if already processed + if [ -n "${processed_blueprints[${abs_blueprint_file}]:-}" ]; then + continue + fi + + # Mark as processed to prevent infinite recursion + processed_blueprints["${abs_blueprint_file}"]=1 + + # Extract localhost dependencies from the blueprint file + # Works for both .containerfile (FROM localhost/...) and .image-bootc (localhost/...) files + local deps="" + deps="$(grep -Eo 'localhost/[a-zA-Z0-9-]+:latest' "${abs_blueprint_file}" | \ + awk -F'localhost/' '{print $2}' | sed 's/:latest//' || true)" + + # Recursively process each dependency + for dep in ${deps}; do + find_dependencies_bootc "${dep}" + done + + # Add this blueprint to the ordered list (dependencies first) + blueprint_order+=("${abs_blueprint_file}") + done +} + +# Recursive function to find dependencies for ostree blueprints +find_dependencies_ostree() { + echo "ERROR: find_dependencies_ostree is not implemented" >&2 + exit 1 +} + +# Process all found images using the appropriate function +for image_name in "${all_images[@]}"; do + if [ -n "${image_name}" ]; then + if [ "${scenario_type}" = "bootc" ]; then + find_dependencies_bootc "${image_name}" + else + find_dependencies_ostree "${image_name}" + fi + fi +done + +# ============================================================================ +# Step 5: Generate Build Commands +# ============================================================================ + +if [ ${#blueprint_order[@]} -eq 0 ]; then + echo "ERROR: No dependencies found for scenario file: ${scenario_file}" >&2 + exit 1 +fi + +# Output build commands in dependency order +for blueprint_file in "${blueprint_order[@]}"; do + if [ "${scenario_type}" = "bootc" ]; then + echo "./test/bin/build_bootc_images.sh --template ${blueprint_file}" + else + echo "./test/bin/build_images.sh -t ${blueprint_file}" + fi +done