Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions .wordlist.txt
Original file line number Diff line number Diff line change
Expand Up @@ -102,7 +102,9 @@ Labeller
labeller
lifecycle
LIF
MachineConfig
MaxUnavailable
MCO
MetricsExporter
MetricsExporterSpec
MPIJob
Expand Down
6 changes: 5 additions & 1 deletion docs/drivers/drivers.md
Original file line number Diff line number Diff line change
Expand Up @@ -38,10 +38,14 @@ spec:
To install the `ionic`, `ionic_rdma`, `tawk_ipc`, and `pds_core` drivers by using AMD Network Operator, please prepare an image registry to store the compiled driver images, then specify corresponding fields in the driver spec of `NetworkConfig`. Please note that user-space drivers are not installed by the Network Operator and need to be installed separately.

```{note}
Some Operating System may contain an inbox `ionic` kernel module. That inbox kernel module could be old versions and affect the installation of desired version out-of-tree kernel module. To blacklist the inbox `ionic` driver please specify `spec.driver.blacklist` as true. After that the worker nodes may need to these to apply the blacklist and avoid the usage of inbox `ionic` kernel module:
Some operating systems may include an in-tree `ionic` kernel module. That module may be an older version and can conflict with the out-of-tree kernel module installed by the operator.

**Kubernetes (Ubuntu):** To blacklist the in-tree `ionic` driver, set `spec.driver.blacklist: true` in the NetworkConfig CR. After that, run the following on each worker node to apply the blacklist:

* sudo update-initramfs -u
* sudo reboot

**OpenShift:** The `spec.driver.blacklist` field is not supported on OpenShift. Instead, use a MachineConfig resource to blacklist the in-tree ionic module. See the [OpenShift Installation Guide](../openshift/installation-guide.md#2-blacklist-in-tree-ionic-driver-recommended) for details.
```

For example:
Expand Down
43 changes: 41 additions & 2 deletions docs/openshift/installation-guide.md
Original file line number Diff line number Diff line change
Expand Up @@ -222,7 +222,46 @@ kubectl patch image.config.openshift.io/cluster --type=merge \

**Note**: This configuration allows all nodes to pull from the specified registry without TLS verification.

### 2. Verify KMM Installation
### 2. Blacklist In-Tree Ionic Driver (Recommended)

OpenShift CoreOS ships with an in-tree `ionic` kernel module that loads at boot and can conflict with the out-of-tree driver installed by the AMD Network Operator. Apply a MachineConfig to blacklist the in-tree module and prevent it from loading at boot. The operator also configures KMM to remove any loaded in-tree module at runtime, but the blacklist prevents the brief window where the in-tree driver is active before KMM intervenes.

Create a `MachineConfig` to blacklist the in-tree `ionic` and `ionic_rdma` modules:

```yaml
apiVersion: machineconfiguration.openshift.io/v1
kind: MachineConfig
metadata:
labels:
machineconfiguration.openshift.io/role: worker
name: ionic-module-blacklist
spec:
config:
ignition:
version: 3.2.0
storage:
files:
- path: "/etc/modprobe.d/ionic-blacklist.conf"
mode: 0644
overwrite: true
contents:
source: "data:text/plain;base64,YmxhY2tsaXN0IGlvbmljCmJsYWNrbGlzdCBpb25pY19yZG1hCg=="
```

Save the above manifest to a file and apply it:

```bash
oc apply -f ionic-module-blacklist.yaml
```

**Note**: Applying a `MachineConfig` will trigger a rolling reboot of the worker nodes managed by the Machine Config Operator (MCO). The base64 content decodes to:

```text
blacklist ionic
blacklist ionic_rdma
```

### 3. Verify KMM Installation

```bash
# Verify KMM is running in openshift-kmm namespace
Expand All @@ -238,7 +277,7 @@ kubectl get deployment -n openshift-kmm

**Troubleshooting**: If KMM exists in multiple namespaces, keep only the one in `openshift-kmm` to avoid conflicts.

### 3. Set Environment Variables
### 4. Set Environment Variables

Set version variables that will be used throughout the deployment:

Expand Down
Loading