-
Notifications
You must be signed in to change notification settings - Fork 13
Add assisted installer deployment method for spoke TNF clusters #51
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Open
gamado
wants to merge
9
commits into
openshift-eng:main
Choose a base branch
from
gamado:assisted
base: main
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
Open
Changes from all commits
Commits
Show all changes
9 commits
Select commit
Hold shift + click to select a range
8b12711
Support Instance type of c5.metal
gamado 7075bb5
Add assisted installer deployment method for spoke TNF clusters
gamado 8719ccc
Fix critical review issues for assisted installer roles
gamado bc768c3
Fix important review issues and add READMEs for assisted installer roles
gamado 6b85d70
Fix spoke package layering and ACM channel detection race conditions
gamado 625462b
Add spoke input validation and remove destructive storage cleanup
gamado 7b4ec76
Fix ipaddr validation to use truthiness instead of 'is not none'
gamado 5dfa87e
Fix ipaddr validation to cast results to bool for assert
gamado 8c0cf33
Fix ipaddr validation: use != false instead of | bool
gamado File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,154 @@ | ||
| --- | ||
| # Deploy a spoke TNF cluster via ACM/assisted installer on an existing hub cluster. | ||
| # | ||
| # Prerequisites: | ||
| # - vars/assisted.yml exists (copy from vars/assisted.yml.template) | ||
| # | ||
| # Usage: | ||
| # make deploy fencing-assisted | ||
|
|
||
| - hosts: metal_machine | ||
| gather_facts: yes | ||
|
|
||
| vars: | ||
| topology: fencing | ||
| interactive_mode: false | ||
| pull_secret_path: /opt/dev-scripts/pull_secret.json | ||
| hub_kubeconfig: "{{ ansible_user_dir }}/auth/kubeconfig" | ||
| method: assisted | ||
| cluster_state_dir: "../aws-hypervisor/instance-data" | ||
| cluster_state_filename: "cluster-vm-state.json" | ||
|
|
||
| vars_files: | ||
| - vars/assisted.yml | ||
|
|
||
| pre_tasks: | ||
| - name: Check that proxy.env exists (hub must be deployed first) | ||
| stat: | ||
| path: "{{ playbook_dir }}/proxy.env" | ||
| delegate_to: localhost | ||
| register: proxy_env_check | ||
|
|
||
| - name: Fail if proxy.env is missing | ||
| fail: | ||
| msg: >- | ||
| proxy.env not found. The hub cluster must be deployed first | ||
| using 'make deploy fencing-ipi'. proxy.env is required for | ||
| cluster access. | ||
| when: not proxy_env_check.stat.exists | ||
|
|
||
| - name: Check that hub kubeconfig exists | ||
| stat: | ||
| path: "{{ ansible_user_dir }}/auth/kubeconfig" | ||
| register: hub_kubeconfig_check | ||
|
|
||
| - name: Fail if hub kubeconfig is missing | ||
| fail: | ||
| msg: >- | ||
| Hub kubeconfig not found at ~/auth/kubeconfig. | ||
| The hub cluster must be deployed first. | ||
| when: not hub_kubeconfig_check.stat.exists | ||
|
|
||
| - name: Preserve hub proxy.env as hub-proxy.env | ||
| copy: | ||
| src: "{{ playbook_dir }}/proxy.env" | ||
| dest: "{{ playbook_dir }}/hub-proxy.env" | ||
| remote_src: no | ||
| backup: no | ||
| delegate_to: localhost | ||
|
|
||
| - name: Get hub release image | ||
| shell: | | ||
| oc get clusterversion version -o jsonpath='{.status.desired.image}' | ||
| register: hub_release_image_raw | ||
| changed_when: false | ||
| environment: | ||
| KUBECONFIG: "{{ hub_kubeconfig }}" | ||
|
|
||
| - name: Get hub OCP version | ||
| shell: | | ||
| oc get clusterversion version -o jsonpath='{.status.desired.version}' | cut -d. -f1-2 | ||
| register: hub_ocp_version_raw | ||
| changed_when: false | ||
| environment: | ||
| KUBECONFIG: "{{ hub_kubeconfig }}" | ||
|
|
||
| - name: Set hub release facts | ||
| set_fact: | ||
| hub_release_image: "{{ hub_release_image_raw.stdout }}" | ||
| hub_ocp_version: "{{ hub_ocp_version_raw.stdout }}" | ||
| effective_release_image: >- | ||
| {{ hub_release_image_raw.stdout if spoke_release_image == 'auto' | ||
| else spoke_release_image }} | ||
| effective_ocp_version: "{{ hub_ocp_version_raw.stdout }}" | ||
|
|
||
| - name: Display assisted installer configuration | ||
| debug: | ||
| msg: | | ||
| Assisted Installer Configuration: | ||
| Hub operator: {{ hub_operator }} | ||
| ACM/MCE channel: {{ acm_channel if hub_operator == 'acm' else mce_channel }} | ||
| Spoke cluster: {{ spoke_cluster_name }}.{{ spoke_base_domain }} | ||
| Spoke release image: {{ spoke_release_image }} | ||
| Spoke VMs: {{ spoke_ctlplanes }}x ({{ spoke_vm_vcpus }} vCPUs, {{ spoke_vm_memory }}MB RAM, {{ spoke_vm_disk_size }}GB disk) | ||
| Spoke network: {{ spoke_network_cidr }} | ||
| API VIP: {{ spoke_api_vip }} | ||
| Ingress VIP: {{ spoke_ingress_vip }} | ||
| Storage method: {{ assisted_storage_method }} | ||
| Force cleanup: {{ force_cleanup }} | ||
|
|
||
| - name: Update cluster state to deploying | ||
| include_role: | ||
| name: common | ||
| tasks_from: cluster-state | ||
| vars: | ||
| cluster_state_phase: 'deploying' | ||
| default_playbook_name: 'assisted-install.yml' | ||
| num_masters: "{{ spoke_ctlplanes }}" | ||
| num_workers: 0 | ||
|
|
||
| roles: | ||
| - role: assisted/acm-install | ||
| - role: assisted/assisted-spoke | ||
|
|
||
| post_tasks: | ||
| - name: Setup proxy access for spoke cluster | ||
| include_role: | ||
| name: proxy-setup | ||
| vars: | ||
| kubeconfig_path: "{{ spoke_kubeconfig_path }}" | ||
| kubeadmin_password_path: "{{ spoke_kubeadmin_password_path }}" | ||
|
|
||
| - name: Update cluster inventory with spoke VMs | ||
| include_role: | ||
| name: common | ||
| tasks_from: update-cluster-inventory | ||
| vars: | ||
| test_cluster_name: "{{ spoke_cluster_name }}" | ||
|
|
||
| - name: Update cluster state to deployed | ||
| include_role: | ||
| name: common | ||
| tasks_from: cluster-state | ||
| vars: | ||
| cluster_state_phase: 'deployed' | ||
| default_playbook_name: 'assisted-install.yml' | ||
| num_masters: "{{ spoke_ctlplanes }}" | ||
| num_workers: 0 | ||
|
|
||
| - name: Display deployment summary | ||
| debug: | ||
| msg: | | ||
| Spoke TNF cluster deployed successfully! | ||
|
|
||
| Spoke credentials: | ||
| Kubeconfig: {{ spoke_kubeconfig_path }} | ||
| Admin password: {{ spoke_kubeadmin_password_path }} | ||
|
|
||
| Access spoke cluster: | ||
| source proxy.env | ||
| KUBECONFIG={{ spoke_kubeconfig_path }} oc get nodes | ||
|
|
||
| Access hub cluster: | ||
| source hub-proxy.env | ||
| KUBECONFIG=~/auth/kubeconfig oc get nodes |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
79 changes: 79 additions & 0 deletions
79
deploy/openshift-clusters/roles/assisted/acm-install/README.md
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,79 @@ | ||
| # acm-install Role | ||
|
|
||
| Installs ACM or MCE operator on a hub cluster and configures the assisted installer service for spoke TNF cluster deployment. | ||
|
|
||
| ## Description | ||
|
|
||
| This role prepares an existing hub OpenShift cluster to deploy spoke TNF clusters via the assisted installer. It: | ||
|
|
||
| 1. Validates hub cluster health and prerequisites | ||
| 2. Provisions hostPath storage for the assisted service | ||
| 3. Installs the ACM or MCE operator (auto-detects channel) | ||
| 4. Creates the AgentServiceConfig with RHCOS ISO auto-extracted from the hub release image | ||
| 5. Enables TNF cluster support in the assisted service | ||
| 6. Configures BMO to watch all namespaces and disables the provisioning network | ||
|
|
||
| ## Requirements | ||
|
|
||
| - A running hub OpenShift cluster (deployed via `make deploy fencing-ipi` or equivalent) | ||
| - Hub kubeconfig accessible at `~/auth/kubeconfig` | ||
| - Pull secret with access to required registries | ||
| - `oc` CLI available on the hypervisor | ||
|
|
||
| ## Role Variables | ||
|
|
||
| ### Configurable Variables (defaults/main.yml) | ||
|
|
||
| - `hub_operator`: Operator to install - `"acm"` or `"mce"` (default: `"acm"`) | ||
| - `acm_channel`: ACM operator channel - `"auto"` detects from packagemanifest (default: `"auto"`) | ||
| - `mce_channel`: MCE operator channel (default: `"auto"`) | ||
| - `assisted_storage_method`: Storage backend - currently only `"hostpath"` (default: `"hostpath"`) | ||
| - `assisted_images_path`: Host directory for ISO images (default: `/var/lib/assisted-images`) | ||
| - `assisted_db_path`: Host directory for database (default: `/var/lib/assisted-db`) | ||
| - `assisted_images_size`: PV size for images (default: `50Gi`) | ||
| - `assisted_db_size`: PV size for database (default: `10Gi`) | ||
| - `assisted_storage_class`: StorageClass name (default: `assisted-service`) | ||
|
|
||
| ### Timeout Variables | ||
|
|
||
| - `acm_csv_timeout`: Operator CSV install timeout in seconds (default: `900`) | ||
| - `multiclusterhub_timeout`: MultiClusterHub readiness timeout (default: `1800`) | ||
| - `assisted_service_timeout`: Assisted service pod readiness timeout (default: `600`) | ||
| - `metal3_stabilize_timeout`: Metal3 pod stabilization timeout after provisioning changes (default: `300`) | ||
|
|
||
| ### Variables Set by Playbook | ||
|
|
||
| These are set in `assisted-install.yml` and passed to the role: | ||
|
|
||
| - `hub_kubeconfig`: Path to hub cluster kubeconfig | ||
| - `pull_secret_path`: Path to pull secret on the hypervisor | ||
| - `hub_release_image`: Hub cluster release image (extracted in playbook pre_tasks) | ||
| - `hub_ocp_version`: Hub OCP version major.minor (extracted in playbook pre_tasks) | ||
| - `effective_release_image`: Release image to use for the spoke (hub image or user override) | ||
|
|
||
| ## Task Flow | ||
|
|
||
| 1. **validate.yml** - Checks hub cluster health, node readiness, and API access | ||
| 2. **storage.yml** - Creates hostPath PVs, StorageClass, and fixes permissions/SELinux on hub nodes | ||
| 3. **install-operator.yml** - Installs ACM/MCE operator subscription, waits for CSV, creates MultiClusterHub | ||
| 4. **agent-service-config.yml** - Extracts RHCOS ISO URL from release image, creates AgentServiceConfig | ||
| 5. **enable-tnf.yml** - Enables TNF support in assisted service configuration | ||
| 6. **enable-watch-all-namespaces.yml** - Patches Provisioning CR to enable BMO in all namespaces | ||
|
|
||
| ## Usage | ||
|
|
||
| This role is not called directly. It is invoked via `assisted-install.yml`: | ||
|
|
||
| ```bash | ||
| make deploy fencing-assisted | ||
| # or | ||
| ansible-playbook assisted-install.yml -i inventory.ini | ||
| ``` | ||
|
|
||
| ## Troubleshooting | ||
|
|
||
| - Check operator CSV status: `oc get csv -n open-cluster-management` | ||
| - Check MultiClusterHub status: `oc get multiclusterhub -n open-cluster-management` | ||
| - Check assisted service pods: `oc get pods -n multicluster-engine -l app=assisted-service` | ||
| - Check AgentServiceConfig: `oc get agentserviceconfig agent -o yaml` | ||
| - Check events: `oc get events -n multicluster-engine --sort-by='.lastTimestamp'` |
25 changes: 25 additions & 0 deletions
25
deploy/openshift-clusters/roles/assisted/acm-install/defaults/main.yml
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,25 @@ | ||
| --- | ||
| # Default variables for acm-install role | ||
|
|
||
| # Hub operator to install: "acm" or "mce" | ||
| hub_operator: acm | ||
|
|
||
| # ACM/MCE channel: "auto" detects from packagemanifest | ||
| acm_channel: "auto" | ||
| mce_channel: "auto" | ||
|
|
||
| # Storage method for assisted service: "hostpath" | ||
| assisted_storage_method: "hostpath" | ||
|
|
||
| # hostPath directories on hub nodes | ||
| assisted_images_path: /var/lib/assisted-images | ||
| assisted_db_path: /var/lib/assisted-db | ||
| assisted_images_size: 50Gi | ||
| assisted_db_size: 10Gi | ||
| assisted_storage_class: assisted-service | ||
|
|
||
| # Timeouts (seconds) | ||
| acm_csv_timeout: 900 | ||
| multiclusterhub_timeout: 1800 | ||
| assisted_service_timeout: 600 | ||
| metal3_stabilize_timeout: 300 |
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.