Fix: Close remaining content gaps

.wordlist.txt

@@ -2,6 +2,7 @@ ACS
 AER
 agfhc
 AGFHC
+allocatable
 agg
 ainic
 AITER
@@ -73,6 +74,7 @@ CNP
 completers
 conf
 config
+cron
 Config
 ConfigFile
 ConnectX
@@ -88,6 +90,7 @@ CX
 CXXFLAGS
 Datacenter
 DCGPU
+DDR
 DLC
 dcqcn
 DCQCN
@@ -99,6 +102,7 @@ DF
 DGEMM
 diags
 DIMM
+DIMMs
 dma
 DMA
 dmesg
@@ -172,6 +176,7 @@ ibv
 ibverbs
 iDRAC
 IET
+initrd
 iface
 ifname
 iLO
@@ -190,6 +195,7 @@ json
 JAX
 JSON
 KASLR
+KiB
 KERNARG
 kx
 Lenovo
@@ -274,6 +280,7 @@ OAM
 OAMs
 OCP
 OEM
+OOM
 OEMs
 OEM’s
 OFED
@@ -305,6 +312,7 @@ perl
 pfc
 Pid
 PID
+POR
 PLDM
 PLX
 Pollara
@@ -401,6 +409,7 @@ UBB
 ubuntu
 UCX
 UEFI
+UMC
 UDEV
 UG
 ulimit

docs/common/kernel-parameters.md

@@ -8,28 +8,42 @@ Configuring the correct kernel command-line parameters is essential for stable o
 
 ## GRUB Configuration Steps
 
-The following steps are for Ubuntu-based systems:
-
+````{tab-set}
+```{tab-item} Ubuntu / Debian
 1. Open `/etc/default/grub` with root privileges.
 2. Locate the line starting with `GRUB_CMDLINE_LINUX`.
 3. Append all required and recommended parameters to this line.
 4. Save the file and apply changes:
 
-   ```bash
-   sudo update-grub
-   ```
+       sudo update-grub
 
 5. Reboot the system for changes to take effect.
-6. Verify the active kernel parameters:
-
-   ```bash
-   cat /proc/cmdline
-   ```
+```
 
-```{note}
-For RHEL-based systems, use the grubby tool instead of editing GRUB directly:
+```{tab-item} RHEL / Rocky / AlmaLinux
+Use the `grubby` tool instead of editing GRUB directly:
 
     sudo grubby --update-kernel=ALL --args="pci=bfsort pci=realloc=off iommu=pt numa_balancing=disable modprobe.blacklist=amdgpu"
+
+Then reboot.
+```
+
+```{tab-item} SLES
+1. Open `/etc/default/grub` with root privileges.
+2. Locate the line starting with `GRUB_CMDLINE_LINUX_DEFAULT`.
+3. Append all required and recommended parameters to this line.
+4. Save the file and apply changes:
+
+       sudo grub2-mkconfig -o /boot/grub2/grub.cfg
+
+5. Reboot the system for changes to take effect.
+```
+````
+
+Verify the active kernel parameters after rebooting:
+
+```bash
+cat /proc/cmdline
 ```
 
 ## Kernel Parameters
@@ -66,7 +80,7 @@ Depending on your specific environment and workload requirements, additional ker
 | `processor.max_cstate=0` | Limits the CPU to the active state, making the CPU more responsive. This setting is highly recommended when prioritizing performance. |
 | `biosdevname=0` | Disables predictable system devices naming. This decreases boot time and has minimal performance increase. |
 | `quiet` | Reduces the number of messages logged to dmesg to only the critical messages. |
-| `transparent_hugepage=always` | Allows applications to automatically allocate huge (2MB) memory pages without application changes. This allows for faster memory allocation and easier memory management at the cost of increased RAM usage. This feature may reduce performance if RAM is constrained. |
+| `transparent_hugepage=always` | **Required on MI300A APU.** On the MI300A, the unified CPU+GPU shared-memory model requires transparent huge pages for correct and performant operation. On discrete OAM systems (MI300X, MI325X, MI350X, MI355X) this is optional: it allows applications to automatically allocate 2 MB pages without application changes, but may reduce performance if RAM is constrained. |
 | `tsc=nowatchdog` | The kernel stops periodically checking TSC accuracy, reducing background activity and optimizing performance. |
 | `nmi_watchdog=0` | Disables the NMI watchdog that detects hard lockups. Disabling this improves high-performance and low-latency workloads. |
 | `nowatchdog` | Disables the kernel's soft and hard lockup watchdog timers, which are normally used to detect and recover from CPU hangs. While this can slightly improve performance by reducing monitoring overhead, it removes a key safeguard against system lockups, making debugging and stability tracking more difficult. |

docs/common/os-tuning.md

@@ -8,6 +8,10 @@ Operating system tuning is essential for achieving optimal performance with AMD
 
 For all BIOS-level optimizations that complement these OS settings, refer to the [BIOS Settings](bios-settings.md) guide.
 
+```{note}
+The AMD Instinct MI300A APU uses a unified CPU+GPU shared-memory architecture that requires additional OS-level configuration not covered in this page. If you are working with MI300A systems, see the [APU-specific OS tuning](../gpus/mi300a.md#apu-specific-os-tuning) section in the MI300A page before proceeding.
+```
+
 ## C-states Configuration
 
 ### C-state Definitions
@@ -20,11 +24,25 @@ For all BIOS-level optimizations that complement these OS settings, refer to the
 
 If C-states must be enabled in BIOS, it's recommended to disable the C2 state to reduce latency, which is particularly important for high-performance networking.
 
-```bash
-# Install cpupower tool
+First, install the `cpupower` tool. The package name varies by distribution:
+
+````{tab-set}
+```{tab-item} Ubuntu
 sudo apt install linux-tools-common
+```
+
+```{tab-item} RHEL
+sudo yum install cpupowerutils
+```
 
-# Disable power-gating on all cores (disable C2 state)
+```{tab-item} SLES
+sudo zypper install cpupower
+```
+````
+
+Then disable power-gating on all cores:
+
+```bash
 cpupower idle-set -d 2
 ```
 
@@ -56,6 +74,28 @@ cat /proc/sys/kernel/numa_balancing
 
 > **Note**: Disabling NUMA balancing should be done cautiously and tested in controlled environments first.
 
+### Persist NUMA balancing setting across kernel updates
+
+Kernel updates can re-enable NUMA balancing. To ensure the setting is always applied at boot, add a cron job for the root user:
+
+```bash
+sudo crontab -e
+```
+
+Add the following line:
+
+```bash
+@reboot sh -c 'echo 0 > /proc/sys/kernel/numa_balancing'
+```
+
+Verify the setting after the next reboot:
+
+```bash
+cat /proc/sys/kernel/numa_balancing
+```
+
+A value of `0` confirms NUMA balancing is disabled.
+
 ## Kernel Parameters for Performance
 
 For recommended kernel parameters, refer to the [Kernel Parameters](kernel-parameters.md) document.
@@ -85,11 +125,24 @@ echo "export HSA_OVERRIDE_CPU_AFFINITY_DEBUG=0" | sudo tee -a /etc/environment
 
 ## IOMMU Configuration for Large Systems
 
-For systems with 256 logical CPU cores or more, configure the kernel for IOMMU passthrough mode:
+For systems with 256 logical CPU cores or more, setting IOMMU to `disabled` in the BIOS can limit the available logical cores to 255. The reason is that the Linux kernel disables X2APIC in this case and falls back to the Advanced Programmable Interrupt Controller (APIC), which can only enumerate a maximum of 255 logical cores.
+
+If SMT is enabled (`CCD/Core/Thread Enablement > SMT Control` set to `enable`), apply the following steps to expose all logical cores:
+
+1. In the server BIOS, set IOMMU to **Enabled**.
+2. Add `iommu=pt` to `GRUB_CMDLINE_LINUX` in `/etc/default/grub`.
+3. Update GRUB and reboot.
+4. Verify IOMMU passthrough mode:
+
+   ```bash
+   dmesg | grep iommu
+   ```
+
+   Expected output includes:
 
-1. **Kernel Parameters**: Add `iommu=pt` to GRUB arguments
-2. **Apply Changes**: Update GRUB and reboot
-3. **Verification**: Check with `dmesg | grep iommu`
+   ```text
+   [    0.000000] Kernel command line: [...] iommu=pt
+   ```
 
 ```{note}
 This kernel setting requires IOMMU to be enabled in the system BIOS. Refer to the [BIOS Settings](bios-settings.md) guide for details.

docs/gpus/mi100.md

@@ -31,7 +31,7 @@ For the most up-to-date information on supported operating systems and distribut
 [ROCm docs](https://rocm.docs.amd.com) is the single source of truth for supported versions, distribution compatibility, and required dependencies for the ROCm toolkit.
 ```
 
-For BIOS, NUMA, and OS-level tuning that applies to all AMD Instinct hosts, see [BIOS settings](../common/bios-settings.md) and [OS tuning](../common/os-tuning.md).
+For general NUMA and OS-level tuning that applies to all AMD Instinct hosts, see [OS tuning](../common/os-tuning.md). MI100-specific BIOS settings are in the [BIOS settings](#bios-settings) section below.
 
 ### GPU identification
 
@@ -50,6 +50,74 @@ Expected output example (4-GPU MI100 hive):
 26:00.0 Display controller: Advanced Micro Devices, Inc. [AMD/ATI] Arcturus GL-XL [Instinct MI100] (rev 01)
 ```
 
+## BIOS settings
+
+For maximum MI100 GPU performance on systems with AMD EPYC™ 7002-series processors (codename "Rome") and AMI System BIOS, the following BIOS settings have been validated. These settings should be set as default values in the system BIOS. Analogous settings for other non-AMI BIOS providers can be configured similarly. On systems with Intel processors, some settings may not apply.
+
+```{note}
+The BIOS setting locations and names may vary by hardware and BIOS vendor. Consult your system documentation for details.
+```
+
+| BIOS setting location | Parameter | Value | Comments |
+|---|---|---|---|
+| Advanced / PCI subsystem settings | Above 4G decoding | Enabled | GPU large BAR support. |
+| AMD CBS / CPU common options | Global C-state control | Auto | Global C-states. |
+| AMD CBS / CPU common options | CCD/Core/Thread enablement | Accept | May be necessary to enable the SMT control menu. |
+| AMD CBS / CPU common options / performance | SMT control | Disable | Set to Auto if the primary application is not compute-bound. |
+| AMD CBS / DF common options / memory addressing | NUMA nodes per socket | NPS 1, 2, or 4 | See [NPS memory configuration](#nps-memory-configuration) below. |
+| AMD CBS / DF common options / memory addressing | Memory interleaving | Auto | Depends on NPS setting. |
+| AMD CBS / DF common options / link | 4-link xGMI max speed | 18 Gbps | Set to highest rate supported by the CPU. |
+| AMD CBS / DF common options / link | 3-link xGMI max speed | 18 Gbps | Set to highest rate supported by the CPU. |
+| AMD CBS / NBIO common options | IOMMU | Disabled | |
+| AMD CBS / NBIO common options | PCIe ten bit tag support | Enabled | |
+| AMD CBS / NBIO common options | Preferred IO | Manual | |
+| AMD CBS / NBIO common options | Preferred IO bus | Use `lspci` to find PCIe device ID | |
+| AMD CBS / NBIO common options | Enhanced Preferred IO mode | Enabled | |
+| AMD CBS / NBIO common options / SMU common options | Determinism control | Manual | |
+| AMD CBS / NBIO common options / SMU common options | Determinism slider | Power | |
+| AMD CBS / NBIO common options / SMU common options | cTDP control | Manual | |
+| AMD CBS / NBIO common options / SMU common options | cTDP | 240 | Value in watts. |
+| AMD CBS / NBIO common options / SMU common options | Package power limit control | Manual | |
+| AMD CBS / NBIO common options / SMU common options | Package power limit | 240 | Value in watts. |
+| AMD CBS / NBIO common options / SMU common options | xGMI link width control | Manual | |
+| AMD CBS / NBIO common options / SMU common options | xGMI force link width | 2 | 0: x2 / 1: x8 / 2: x16 |
+| AMD CBS / NBIO common options / SMU common options | xGMI force link width control | Force | |
+| AMD CBS / NBIO common options / SMU common options | APBDIS | 1 | Disable DF P-states. |
+| AMD CBS / NBIO common options / SMU common options | DF C-states | Auto | |
+| AMD CBS / NBIO common options / SMU common options | Fixed SOC P-state | P0 | |
+| AMD CBS / UMC common options / DDR4 common options | Enforce POR | Accept | |
+| AMD CBS / UMC common options / DDR4 common options / Enforce POR | Overclock | Enabled | |
+| AMD CBS / UMC common options / DDR4 common options / Enforce POR | Memory clock speed | 1600 MHz | Set to max speed if using 3200 MHz DIMMs. |
+| AMD CBS / UMC common options / DDR4 common options / DRAM controller configuration / DRAM power options | Power down enable | Disabled | RAM power down. |
+| AMD CBS / security | TSME | Disabled | Memory encryption. |
+
+### NBIO link clock frequency
+
+The NBIOs (4× per AMD EPYC™ processor) are the serializers/deserializers (SerDes) that convert and prepare I/O signals for the processor's 128 external I/O interface lanes (32 per NBIO). The NBIO link clock frequency (LCLK) controls the speed of the internal bus connecting NBIO silicon to the data fabric. All data between the processor and PCIe lanes flows through the data fabric at this frequency, so it must be forced to the maximum for optimal PCIe performance.
+
+For AMD EPYC™ 7002-series processors, this cannot be set via BIOS alone. The **AMD-IOPM-UTIL** must be run at every server boot to disable Dynamic Power Management for all PCIe root complexes and NBIOs and lock them into the highest performance mode. See [AMD-IOPM-UTIL](#amd-iopm-util) below.
+
+### NPS memory configuration
+
+For the number of NUMA nodes per socket (NPS), follow the guidance of the [HPC Tuning Guide for AMD EPYC™ 7002 Series Processors](https://docs.amd.com/v/u/en-US/amd-epyc-7002-tg-hpc-56827) for the optimal host configuration.
+
+- **NPS1**: Bidirectional copy bandwidth between host memory and GPU memory may be up to ~16% higher than with NPS4. Recommended for applications not optimized for NUMA locality.
+- **NPS4**: Recommended for memory-bandwidth-sensitive applications using MPI.
+
+### AMD-IOPM-UTIL
+
+The AMD I/O Power Management Utility (AMD-IOPM-UTIL) disables Dynamic Power Management (DPM) for all PCIe root complexes on AMD EPYC™ 7002-series processors and locks the NBIO logic into the highest performance operational mode. This is required for MI100 systems to ensure maximum LCLK frequency.
+
+Disabling I/O DPM reduces latency and improves throughput for low-bandwidth PCIe messages, benefiting InfiniBand NICs, GPUs, and other bursty PCIe devices.
+
+```{note}
+The effects of AMD-IOPM-UTIL do not persist across reboots. No firmware settings need to change when using this utility. The "Preferred IO" and "Enhanced Preferred IO" BIOS settings should remain enabled.
+```
+
+The recommended method is to create a one-shot `systemd` service unit that runs the utility at boot. The installer packages from the [AMD I/O Power Management Utility page](https://developer.amd.com/iopm-utility/) create and enable this service unit automatically. The service runs in one-shot mode, so `systemctl status` will show **inactive** after a successful run — this is expected. A **failed** status means the utility did not run correctly.
+
+To undo the effects, disable the service with `systemctl disable` and reboot.
+
 ## Acceptance criteria
 
 The MI100 system acceptance process validates that the platform is correctly configured, stable, and performing to expectations. Follow the sequence: Prerequisites → Basic Health Checks → System Validation → Performance Benchmarks.
@@ -69,7 +137,7 @@ Ensure all system requirements are met before proceeding with validation. See th
 
 - ✅ Supported operating system version installed
 - ✅ Compatible ROCm version installed
-- ✅ BIOS configured per [BIOS settings](../common/bios-settings.md), with MI100-specific values per platform vendor
+- ✅ BIOS configured per [MI100 BIOS settings](#bios-settings) (EPYC 7002-specific table above)
 - ✅ Required kernel parameters present: `pci=realloc=off`, `pci=bfsort`, `iommu=pt`, and `amd_iommu=on` (or `intel_iommu=on` on Intel hosts) — see [Kernel Parameters](../common/kernel-parameters.md)
 - ✅ Minimum 256G system memory available
 - ✅ Latest applicable firmware applied consistently across nodes