s390x: do not emit ACPI in domain XML (RHEL 9 QEMU rejects define)

[ibm-adarsh]

Summary

On RHEL 9 s390x hypervisors (QEMU machine type s390-ccw-virtio-rhel9.*), machine-api / cluster-api-provider-libvirt fails to create worker domains because the provider always emits <acpi/> in the guest domain XML. libvirt rejects the domain at define time with:

unsupported configuration: machine type 's390-ccw-virtio-rhel9.8.0' does not support ACPI

s390(x) never supported ACPI. Older libvirt/QEMU stacks often ignored <acpi/>; newer stacks (libvirt ≥ ~9.2 with QEMU that reports ACPI support per machine) reject it. OpenShift IPI libvirt installs on IBM Z therefore get masters (created via openshift-install/terraform, which can be XSLT-patched in CI) but workers stuck forever in Provisioning / InstanceNotCreated.

Exact offending code (revalidated)

File: pkg/cloud/libvirt/client/domain.go

Function: newDomainDef() (starts at L38)

Line that causes the failure: L57 — unconditional ACPI feature:

55:		Features: &libvirtxml.DomainFeatureList{
56:			PAE:  &libvirtxml.DomainFeature{},
57:			ACPI: &libvirtxml.DomainFeature{},   // <-- emits <acpi/> for every guest, including s390x
58:			APIC: &libvirtxml.DomainFeatureAPIC{},
59:		},

Permalink (main @ 63c1b6342e27):
https://github.com/openshift/cluster-api-provider-libvirt/blob/63c1b6342e27/pkg/cloud/libvirt/client/domain.go#L57

Same line on release-4.15 (@ 416999a8de40):
https://github.com/openshift/cluster-api-provider-libvirt/blob/416999a8de40/pkg/cloud/libvirt/client/domain.go#L57

Call path: CreateDomain → newDomainDefForConnection() (L175–L176) → newDomainDef() → domain XML includes <features><acpi/></features> → libvirt DomainDefineXML fails on s390x RHEL 9.

There is no arch check around this feature list (unlike ignition disk handling later in the same file, which already special-cases s390/ppc64).

Environment

Item	Value
Arch	s390x
Platform	libvirt IPI on IBM Z VPN OZ LPARs (phc-cicd.cis.ibm.net)
Hypervisor example	lnxocp14.phc-cicd.cis.ibm.net
QEMU machine type	s390-ccw-virtio-rhel9.8.0
OCP stream	4.15.0-0.nightly-s390x (also affects newer releases on RHEL 9 s390 hosts)
Provider	openshift/cluster-api-provider-libvirt (machine-controller in openshift-machine-api)
Reproducing job	rehearse … ocp-e2e-ovn-remote-libvirt-s390x #2082378715693584384
Related release PR	openshift/release#77414

Actual error (machine-controller)

From openshift-machine-api machine-controller logs while creating workers:

I0729 08:33:33.602193       1 client.go:267] Creating libvirt domain at qemu+tcp://lnxocp14.phc-cicd.cis.ibm.net/system
I0729 08:33:33.602919       1 client.go:274] Creating libvirt domain with XML:
  <domain type="kvm">
      <name>libvirt-s390x-oz-3-1-8gptb-worker-0-mc4q5</name>
      <memory unit="MiB">20480</memory>
      <vcpu>4</vcpu>
      <os>
          <type arch="s390x" machine="s390-ccw-virtio-rhel9.8.0">hvm</type>
      </os>
      <features>
          <pae></pae>
          <acpi></acpi>
          <apic></apic>
      </features>
      ...
  </domain>
E0729 08:33:33.609596       1 actuator.go:107] Machine error: error creating domain error defining libvirt domain: virError(Code=67, Domain=10, Message='unsupported configuration: machine type 's390-ccw-virtio-rhel9.8.0' does not support ACPI')
E0729 08:33:33.609606       1 actuator.go:51] libvirt-s390x-oz-3-1-8gptb-worker-0-mc4q5: error creating libvirt machine: error creating domain error defining libvirt domain: virError(Code=67, Domain=10, Message='unsupported configuration: machine type 's390-ccw-virtio-rhel9.8.0' does not support ACPI')
W0729 08:33:33.609769       1 controller.go:382] libvirt-s390x-oz-3-1-8gptb-worker-0-mc4q5: failed to create machine: ... does not support ACPI

This retries continuously (same CreateError for both workers).

Cluster / Machine status

• Nodes: 3 masters Ready=True; 0 workers
• Machines:
  • masters: phase=Running
  • workers: phase=Provisioning, InstanceExists=False, reason InstanceNotCreated
• MachineSet: replicas=2, readyReplicas unset
• Events: FailedCreate: CreateError on worker Machines (many times)

Cascading install failure (symptoms, not root cause)

Because no workers join, ingress/router cannot schedule (master taint), and install fails waiting for operators, e.g.:

level=error msg=Cluster operator machine-api Degraded is True with SyncingFailed: ... minimum worker replica count (2) not yet met: current running replicas 0, waiting for [libvirt-s390x-oz-3-1-8gptb-worker-0-mc4q5 libvirt-s390x-oz-3-1-8gptb-worker-0-qqmgz]
level=error msg=ReadyIngressNodesAvailable: Authentication requires functional ingress which requires at least one schedulable and ready node. Got 0 worker nodes, 3 master nodes...
level=error msg=OAuthServerDeploymentDegraded: waiting for the oauth-openshift route to contain an admitted ingress: no admitted ingress for route oauth-openshift in namespace openshift-authentication
level=error msg=Cluster operator ingress Degraded ... Pod "router-default-..." cannot be scheduled: 0/3 nodes are available: 3 node(s) had untolerated taint {node-role.kubernetes.io/master: }
level=error msg=failed to initialize the cluster: Cluster operators authentication, console, image-registry, ingress, machine-api, monitoring are not available

Root cause remains: workers never defined due to ACPI from L57 above.

Expected behavior

On s390x (and any arch/machine where QEMU reports ACPI unsupported):

• Do not emit <acpi/> (and likely not x86-centric <pae/> / <apic/> either, unless known-valid for that arch).
• Worker Machines should define/start successfully on RHEL 9 s390x libvirt hosts.

Proposed fix

1. In newDomainDef() (pkg/cloud/libvirt/client/domain.go L55–L59), omit ACPI (and other unsupported features) when arch is s390/s390x.
   • Minimal change: do not set ACPI: &libvirtxml.DomainFeature{} on s390x (line L57).
2. Optionally make features conditional on arch more generally:
   • s390x: no ACPI/PAE/APIC
   • ppc64le: review similarly
   • x86_64/aarch64: keep current behavior
3. Note: newDomainDef() currently has no arch argument; either pass arch into it / newDomainDefForConnection, or clear Features.ACPI after arch is known in domainDefInit / Create path.
4. Add a unit test that asserts s390x domain XML does not contain <acpi.
5. Document that RHEL 9+ s390x QEMU rejects ACPI at define time.

Related / out of scope for this repo but same class of bug

openshift-install’s bundled terraform-provider-libvirt also always enables ACPI for bootstrap/master domains. Multiarch CI currently works around that with an install-step XSLT (IPI_LIBVIRT_S390X_ACPI_XSLT_PATCH). That does not help workers, which are created by this provider after bootstrap.

A durable fix is:

1. This issue (machine-api / CAPL) — workers — L57 in newDomainDef()
2. Matching change in terraform-provider-libvirt / installer — masters/bootstrap

Workaround today (CI / ops only)

• Host: custom libvirt patch to strip ACPI on s390 define (not a qemu hook), or
• CI hybrid: WORKER_REPLICAS=0 + day-2 virsh workers with ACPI-free XML

Neither replaces fixing the provider.

Artifacts

• Prow job: https://prow.ci.openshift.org/view/gs/test-platform-results/pr-logs/pull/openshift_release/77414/rehearse-77414-periodic-ci-openshift-multiarch-main-nightly-4.15-ocp-e2e-ovn-remote-libvirt-s390x/2082378715693584384
• machine-controller log (excerpt above) from gather-extra pods under that job
• Cluster profile / lease: libvirt-s390x-vpn-oz / libvirt-s390x-oz-3-1

Acceptance criteria

• s390x domain XML generated by CAPL does not include <acpi/> (i.e. L57 not applied for s390x)
• virDomainDefineXML / provider Create succeeds against RHEL 9 s390x QEMU (s390-ccw-virtio-rhel9.*)
• OpenShift libvirt IPI on IBM Z can bring workers to Running / Ready via machine-api without host ACPI hacks
• Unit/regression coverage for s390x feature flags