From 8c1e6d5ca20e7a4786f23f1cadc44bd3626ac239 Mon Sep 17 00:00:00 2001 From: Jan Stephan Date: Wed, 24 Jun 2026 16:10:38 +0200 Subject: [PATCH] fix: close remaining content gaps * System optimization guide: Close remaining content gaps Signed-off-by: Jan Stephan * Fix linter issues and update wordlist Signed-off-by: Jan Stephan --------- Signed-off-by: Jan Stephan --- .wordlist.txt | 9 ++++ docs/common/kernel-parameters.md | 40 +++++++++----- docs/common/os-tuning.md | 67 ++++++++++++++++++++--- docs/gpus/mi100.md | 72 ++++++++++++++++++++++++- docs/gpus/mi210.md | 54 ++++++++++++++++++- docs/gpus/mi250.md | 55 ++++++++++++++++++- docs/gpus/mi300a.md | 91 ++++++++++++++++++++++++++++++-- 7 files changed, 359 insertions(+), 29 deletions(-) diff --git a/.wordlist.txt b/.wordlist.txt index 3587820..735fb50 100644 --- a/.wordlist.txt +++ b/.wordlist.txt @@ -2,6 +2,7 @@ ACS AER agfhc AGFHC +allocatable agg ainic AITER @@ -73,6 +74,7 @@ CNP completers conf config +cron Config ConfigFile ConnectX @@ -88,6 +90,7 @@ CX CXXFLAGS Datacenter DCGPU +DDR DLC dcqcn DCQCN @@ -99,6 +102,7 @@ DF DGEMM diags DIMM +DIMMs dma DMA dmesg @@ -172,6 +176,7 @@ ibv ibverbs iDRAC IET +initrd iface ifname iLO @@ -190,6 +195,7 @@ json JAX JSON KASLR +KiB KERNARG kx Lenovo @@ -274,6 +280,7 @@ OAM OAMs OCP OEM +OOM OEMs OEM’s OFED @@ -305,6 +312,7 @@ perl pfc Pid PID +POR PLDM PLX Pollara @@ -401,6 +409,7 @@ UBB ubuntu UCX UEFI +UMC UDEV UG ulimit diff --git a/docs/common/kernel-parameters.md b/docs/common/kernel-parameters.md index 61a981a..a3fd469 100644 --- a/docs/common/kernel-parameters.md +++ b/docs/common/kernel-parameters.md @@ -8,28 +8,42 @@ Configuring the correct kernel command-line parameters is essential for stable o ## GRUB Configuration Steps -The following steps are for Ubuntu-based systems: - +````{tab-set} +```{tab-item} Ubuntu / Debian 1. Open `/etc/default/grub` with root privileges. 2. Locate the line starting with `GRUB_CMDLINE_LINUX`. 3. Append all required and recommended parameters to this line. 4. Save the file and apply changes: - ```bash - sudo update-grub - ``` + sudo update-grub 5. Reboot the system for changes to take effect. -6. Verify the active kernel parameters: - - ```bash - cat /proc/cmdline - ``` +``` -```{note} -For RHEL-based systems, use the grubby tool instead of editing GRUB directly: +```{tab-item} RHEL / Rocky / AlmaLinux +Use the `grubby` tool instead of editing GRUB directly: sudo grubby --update-kernel=ALL --args="pci=bfsort pci=realloc=off iommu=pt numa_balancing=disable modprobe.blacklist=amdgpu" + +Then reboot. +``` + +```{tab-item} SLES +1. Open `/etc/default/grub` with root privileges. +2. Locate the line starting with `GRUB_CMDLINE_LINUX_DEFAULT`. +3. Append all required and recommended parameters to this line. +4. Save the file and apply changes: + + sudo grub2-mkconfig -o /boot/grub2/grub.cfg + +5. Reboot the system for changes to take effect. +``` +```` + +Verify the active kernel parameters after rebooting: + +```bash +cat /proc/cmdline ``` ## Kernel Parameters @@ -66,7 +80,7 @@ Depending on your specific environment and workload requirements, additional ker | `processor.max_cstate=0` | Limits the CPU to the active state, making the CPU more responsive. This setting is highly recommended when prioritizing performance. | | `biosdevname=0` | Disables predictable system devices naming. This decreases boot time and has minimal performance increase. | | `quiet` | Reduces the number of messages logged to dmesg to only the critical messages. | -| `transparent_hugepage=always` | Allows applications to automatically allocate huge (2MB) memory pages without application changes. This allows for faster memory allocation and easier memory management at the cost of increased RAM usage. This feature may reduce performance if RAM is constrained. | +| `transparent_hugepage=always` | **Required on MI300A APU.** On the MI300A, the unified CPU+GPU shared-memory model requires transparent huge pages for correct and performant operation. On discrete OAM systems (MI300X, MI325X, MI350X, MI355X) this is optional: it allows applications to automatically allocate 2 MB pages without application changes, but may reduce performance if RAM is constrained. | | `tsc=nowatchdog` | The kernel stops periodically checking TSC accuracy, reducing background activity and optimizing performance. | | `nmi_watchdog=0` | Disables the NMI watchdog that detects hard lockups. Disabling this improves high-performance and low-latency workloads. | | `nowatchdog` | Disables the kernel's soft and hard lockup watchdog timers, which are normally used to detect and recover from CPU hangs. While this can slightly improve performance by reducing monitoring overhead, it removes a key safeguard against system lockups, making debugging and stability tracking more difficult. | diff --git a/docs/common/os-tuning.md b/docs/common/os-tuning.md index f5d4b19..00d8740 100644 --- a/docs/common/os-tuning.md +++ b/docs/common/os-tuning.md @@ -8,6 +8,10 @@ Operating system tuning is essential for achieving optimal performance with AMD For all BIOS-level optimizations that complement these OS settings, refer to the [BIOS Settings](bios-settings.md) guide. +```{note} +The AMD Instinct MI300A APU uses a unified CPU+GPU shared-memory architecture that requires additional OS-level configuration not covered in this page. If you are working with MI300A systems, see the [APU-specific OS tuning](../gpus/mi300a.md#apu-specific-os-tuning) section in the MI300A page before proceeding. +``` + ## C-states Configuration ### C-state Definitions @@ -20,11 +24,25 @@ For all BIOS-level optimizations that complement these OS settings, refer to the If C-states must be enabled in BIOS, it's recommended to disable the C2 state to reduce latency, which is particularly important for high-performance networking. -```bash -# Install cpupower tool +First, install the `cpupower` tool. The package name varies by distribution: + +````{tab-set} +```{tab-item} Ubuntu sudo apt install linux-tools-common +``` + +```{tab-item} RHEL +sudo yum install cpupowerutils +``` -# Disable power-gating on all cores (disable C2 state) +```{tab-item} SLES +sudo zypper install cpupower +``` +```` + +Then disable power-gating on all cores: + +```bash cpupower idle-set -d 2 ``` @@ -56,6 +74,28 @@ cat /proc/sys/kernel/numa_balancing > **Note**: Disabling NUMA balancing should be done cautiously and tested in controlled environments first. +### Persist NUMA balancing setting across kernel updates + +Kernel updates can re-enable NUMA balancing. To ensure the setting is always applied at boot, add a cron job for the root user: + +```bash +sudo crontab -e +``` + +Add the following line: + +```bash +@reboot sh -c 'echo 0 > /proc/sys/kernel/numa_balancing' +``` + +Verify the setting after the next reboot: + +```bash +cat /proc/sys/kernel/numa_balancing +``` + +A value of `0` confirms NUMA balancing is disabled. + ## Kernel Parameters for Performance For recommended kernel parameters, refer to the [Kernel Parameters](kernel-parameters.md) document. @@ -85,11 +125,24 @@ echo "export HSA_OVERRIDE_CPU_AFFINITY_DEBUG=0" | sudo tee -a /etc/environment ## IOMMU Configuration for Large Systems -For systems with 256 logical CPU cores or more, configure the kernel for IOMMU passthrough mode: +For systems with 256 logical CPU cores or more, setting IOMMU to `disabled` in the BIOS can limit the available logical cores to 255. The reason is that the Linux kernel disables X2APIC in this case and falls back to the Advanced Programmable Interrupt Controller (APIC), which can only enumerate a maximum of 255 logical cores. + +If SMT is enabled (`CCD/Core/Thread Enablement > SMT Control` set to `enable`), apply the following steps to expose all logical cores: + +1. In the server BIOS, set IOMMU to **Enabled**. +2. Add `iommu=pt` to `GRUB_CMDLINE_LINUX` in `/etc/default/grub`. +3. Update GRUB and reboot. +4. Verify IOMMU passthrough mode: + + ```bash + dmesg | grep iommu + ``` + + Expected output includes: -1. **Kernel Parameters**: Add `iommu=pt` to GRUB arguments -2. **Apply Changes**: Update GRUB and reboot -3. **Verification**: Check with `dmesg | grep iommu` + ```text + [ 0.000000] Kernel command line: [...] iommu=pt + ``` ```{note} This kernel setting requires IOMMU to be enabled in the system BIOS. Refer to the [BIOS Settings](bios-settings.md) guide for details. diff --git a/docs/gpus/mi100.md b/docs/gpus/mi100.md index d6b64ee..da62617 100644 --- a/docs/gpus/mi100.md +++ b/docs/gpus/mi100.md @@ -31,7 +31,7 @@ For the most up-to-date information on supported operating systems and distribut [ROCm docs](https://rocm.docs.amd.com) is the single source of truth for supported versions, distribution compatibility, and required dependencies for the ROCm toolkit. ``` -For BIOS, NUMA, and OS-level tuning that applies to all AMD Instinct hosts, see [BIOS settings](../common/bios-settings.md) and [OS tuning](../common/os-tuning.md). +For general NUMA and OS-level tuning that applies to all AMD Instinct hosts, see [OS tuning](../common/os-tuning.md). MI100-specific BIOS settings are in the [BIOS settings](#bios-settings) section below. ### GPU identification @@ -50,6 +50,74 @@ Expected output example (4-GPU MI100 hive): 26:00.0 Display controller: Advanced Micro Devices, Inc. [AMD/ATI] Arcturus GL-XL [Instinct MI100] (rev 01) ``` +## BIOS settings + +For maximum MI100 GPU performance on systems with AMD EPYC™ 7002-series processors (codename "Rome") and AMI System BIOS, the following BIOS settings have been validated. These settings should be set as default values in the system BIOS. Analogous settings for other non-AMI BIOS providers can be configured similarly. On systems with Intel processors, some settings may not apply. + +```{note} +The BIOS setting locations and names may vary by hardware and BIOS vendor. Consult your system documentation for details. +``` + +| BIOS setting location | Parameter | Value | Comments | +|---|---|---|---| +| Advanced / PCI subsystem settings | Above 4G decoding | Enabled | GPU large BAR support. | +| AMD CBS / CPU common options | Global C-state control | Auto | Global C-states. | +| AMD CBS / CPU common options | CCD/Core/Thread enablement | Accept | May be necessary to enable the SMT control menu. | +| AMD CBS / CPU common options / performance | SMT control | Disable | Set to Auto if the primary application is not compute-bound. | +| AMD CBS / DF common options / memory addressing | NUMA nodes per socket | NPS 1, 2, or 4 | See [NPS memory configuration](#nps-memory-configuration) below. | +| AMD CBS / DF common options / memory addressing | Memory interleaving | Auto | Depends on NPS setting. | +| AMD CBS / DF common options / link | 4-link xGMI max speed | 18 Gbps | Set to highest rate supported by the CPU. | +| AMD CBS / DF common options / link | 3-link xGMI max speed | 18 Gbps | Set to highest rate supported by the CPU. | +| AMD CBS / NBIO common options | IOMMU | Disabled | | +| AMD CBS / NBIO common options | PCIe ten bit tag support | Enabled | | +| AMD CBS / NBIO common options | Preferred IO | Manual | | +| AMD CBS / NBIO common options | Preferred IO bus | Use `lspci` to find PCIe device ID | | +| AMD CBS / NBIO common options | Enhanced Preferred IO mode | Enabled | | +| AMD CBS / NBIO common options / SMU common options | Determinism control | Manual | | +| AMD CBS / NBIO common options / SMU common options | Determinism slider | Power | | +| AMD CBS / NBIO common options / SMU common options | cTDP control | Manual | | +| AMD CBS / NBIO common options / SMU common options | cTDP | 240 | Value in watts. | +| AMD CBS / NBIO common options / SMU common options | Package power limit control | Manual | | +| AMD CBS / NBIO common options / SMU common options | Package power limit | 240 | Value in watts. | +| AMD CBS / NBIO common options / SMU common options | xGMI link width control | Manual | | +| AMD CBS / NBIO common options / SMU common options | xGMI force link width | 2 | 0: x2 / 1: x8 / 2: x16 | +| AMD CBS / NBIO common options / SMU common options | xGMI force link width control | Force | | +| AMD CBS / NBIO common options / SMU common options | APBDIS | 1 | Disable DF P-states. | +| AMD CBS / NBIO common options / SMU common options | DF C-states | Auto | | +| AMD CBS / NBIO common options / SMU common options | Fixed SOC P-state | P0 | | +| AMD CBS / UMC common options / DDR4 common options | Enforce POR | Accept | | +| AMD CBS / UMC common options / DDR4 common options / Enforce POR | Overclock | Enabled | | +| AMD CBS / UMC common options / DDR4 common options / Enforce POR | Memory clock speed | 1600 MHz | Set to max speed if using 3200 MHz DIMMs. | +| AMD CBS / UMC common options / DDR4 common options / DRAM controller configuration / DRAM power options | Power down enable | Disabled | RAM power down. | +| AMD CBS / security | TSME | Disabled | Memory encryption. | + +### NBIO link clock frequency + +The NBIOs (4× per AMD EPYC™ processor) are the serializers/deserializers (SerDes) that convert and prepare I/O signals for the processor's 128 external I/O interface lanes (32 per NBIO). The NBIO link clock frequency (LCLK) controls the speed of the internal bus connecting NBIO silicon to the data fabric. All data between the processor and PCIe lanes flows through the data fabric at this frequency, so it must be forced to the maximum for optimal PCIe performance. + +For AMD EPYC™ 7002-series processors, this cannot be set via BIOS alone. The **AMD-IOPM-UTIL** must be run at every server boot to disable Dynamic Power Management for all PCIe root complexes and NBIOs and lock them into the highest performance mode. See [AMD-IOPM-UTIL](#amd-iopm-util) below. + +### NPS memory configuration + +For the number of NUMA nodes per socket (NPS), follow the guidance of the [HPC Tuning Guide for AMD EPYC™ 7002 Series Processors](https://docs.amd.com/v/u/en-US/amd-epyc-7002-tg-hpc-56827) for the optimal host configuration. + +- **NPS1**: Bidirectional copy bandwidth between host memory and GPU memory may be up to ~16% higher than with NPS4. Recommended for applications not optimized for NUMA locality. +- **NPS4**: Recommended for memory-bandwidth-sensitive applications using MPI. + +### AMD-IOPM-UTIL + +The AMD I/O Power Management Utility (AMD-IOPM-UTIL) disables Dynamic Power Management (DPM) for all PCIe root complexes on AMD EPYC™ 7002-series processors and locks the NBIO logic into the highest performance operational mode. This is required for MI100 systems to ensure maximum LCLK frequency. + +Disabling I/O DPM reduces latency and improves throughput for low-bandwidth PCIe messages, benefiting InfiniBand NICs, GPUs, and other bursty PCIe devices. + +```{note} +The effects of AMD-IOPM-UTIL do not persist across reboots. No firmware settings need to change when using this utility. The "Preferred IO" and "Enhanced Preferred IO" BIOS settings should remain enabled. +``` + +The recommended method is to create a one-shot `systemd` service unit that runs the utility at boot. The installer packages from the [AMD I/O Power Management Utility page](https://developer.amd.com/iopm-utility/) create and enable this service unit automatically. The service runs in one-shot mode, so `systemctl status` will show **inactive** after a successful run — this is expected. A **failed** status means the utility did not run correctly. + +To undo the effects, disable the service with `systemctl disable` and reboot. + ## Acceptance criteria The MI100 system acceptance process validates that the platform is correctly configured, stable, and performing to expectations. Follow the sequence: Prerequisites → Basic Health Checks → System Validation → Performance Benchmarks. @@ -69,7 +137,7 @@ Ensure all system requirements are met before proceeding with validation. See th - ✅ Supported operating system version installed - ✅ Compatible ROCm version installed -- ✅ BIOS configured per [BIOS settings](../common/bios-settings.md), with MI100-specific values per platform vendor +- ✅ BIOS configured per [MI100 BIOS settings](#bios-settings) (EPYC 7002-specific table above) - ✅ Required kernel parameters present: `pci=realloc=off`, `pci=bfsort`, `iommu=pt`, and `amd_iommu=on` (or `intel_iommu=on` on Intel hosts) — see [Kernel Parameters](../common/kernel-parameters.md) - ✅ Minimum 256G system memory available - ✅ Latest applicable firmware applied consistently across nodes diff --git a/docs/gpus/mi210.md b/docs/gpus/mi210.md index cdb4dc9..e1aa2bd 100644 --- a/docs/gpus/mi210.md +++ b/docs/gpus/mi210.md @@ -30,7 +30,7 @@ For the most up-to-date information on supported operating systems and distribut [ROCm docs](https://rocm.docs.amd.com) is the single source of truth for supported versions, distribution compatibility, and required dependencies for the ROCm toolkit. ``` -For BIOS, NUMA, and OS-level tuning that applies to all AMD Instinct hosts, see [BIOS settings](../common/bios-settings.md) and [OS tuning](../common/os-tuning.md). MI210 systems share the general OS and IOMMU guidance documented for other CDNA 2 platforms but might differ in BIOS power and xGMI topology settings; consult your platform vendor's BIOS guide for MI210-specific values. +For general NUMA and OS-level tuning that applies to all AMD Instinct hosts, see [OS tuning](../common/os-tuning.md). MI210 systems share the CDNA 2 BIOS baseline with MI250 (EPYC 7003-series); MI210-specific values are in the [BIOS settings](#bios-settings) section below. Power limits and xGMI topology entries may differ from MI250 — consult your platform vendor's BIOS guide for MI210-specific values. ### GPU identification @@ -53,6 +53,56 @@ c3:00.0 Display controller: Advanced Micro Devices, Inc. [AMD/ATI] Aldebaran/MI2 e3:00.0 Display controller: Advanced Micro Devices, Inc. [AMD/ATI] Aldebaran/MI200 [Instinct MI210] (rev 02) ``` +## BIOS settings + +For maximum MI210 GPU performance on systems with AMD EPYC™ 7003-series processors (codename "Milan") and AMI System BIOS, the following BIOS settings have been validated. These settings should be set as default values in the system BIOS. Analogous settings for other non-AMI BIOS providers can be configured similarly. On systems with Intel processors, some settings may not apply. + +```{note} +The BIOS setting locations and names may vary by hardware and BIOS vendor. Consult your system documentation for details. +``` + +| BIOS setting location | Parameter | Value | Comments | +|---|---|---|---| +| Advanced / PCI subsystem settings | Above 4G decoding | Enabled | GPU large BAR support. | +| AMD CBS / CPU common options | Global C-state control | Auto | Global C-states. | +| AMD CBS / CPU common options | CCD/Core/Thread enablement | Accept | May be necessary to enable the SMT control menu. | +| AMD CBS / CPU common options / performance | SMT control | Disable | Set to Auto if the primary application is not compute-bound. | +| AMD CBS / DF common options / memory addressing | NUMA nodes per socket | NPS 1, 2, or 4 | See [NPS memory configuration](#nps-memory-configuration) below. | +| AMD CBS / DF common options / memory addressing | Memory interleaving | Auto | Depends on NPS setting. | +| AMD CBS / DF common options / link | 4-link xGMI max speed | 18 Gbps | Set to highest rate supported by the CPU. | +| AMD CBS / NBIO common options | IOMMU | Disabled | | +| AMD CBS / NBIO common options | PCIe ten bit tag support | Auto | | +| AMD CBS / NBIO common options | Preferred IO | Bus | | +| AMD CBS / NBIO common options | Preferred IO bus | Use `lspci` to find PCIe device ID | | +| AMD CBS / NBIO common options | Enhanced Preferred IO mode | Enabled | | +| AMD CBS / NBIO common options / SMU common options | Determinism control | Manual | | +| AMD CBS / NBIO common options / SMU common options | Determinism slider | Power | | +| AMD CBS / NBIO common options / SMU common options | cTDP control | Manual | Set cTDP to the maximum supported by the installed CPU. | +| AMD CBS / NBIO common options / SMU common options | cTDP | Varies by CPU | Value in watts. Consult your platform vendor. | +| AMD CBS / NBIO common options / SMU common options | Package power limit control | Manual | Set to the maximum supported by the installed CPU. | +| AMD CBS / NBIO common options / SMU common options | Package power limit | Varies by CPU | Value in watts. Consult your platform vendor. | +| AMD CBS / NBIO common options / SMU common options | xGMI link width control | Manual | | +| AMD CBS / NBIO common options / SMU common options | xGMI force link width | 2 | 0: x2 / 1: x8 / 2: x16 | +| AMD CBS / NBIO common options / SMU common options | xGMI force link width control | Force | | +| AMD CBS / NBIO common options / SMU common options | APBDIS | 1 | Disable DF P-states. | +| AMD CBS / NBIO common options / SMU common options | DF C-states | Enabled | | +| AMD CBS / NBIO common options / SMU common options | Fixed SOC P-state | P0 | | +| AMD CBS / UMC common options / DDR4 common options | Enforce POR | Accept | | +| AMD CBS / UMC common options / DDR4 common options / Enforce POR | Overclock | Enabled | | +| AMD CBS / UMC common options / DDR4 common options / Enforce POR | Memory clock speed | 1600 MHz | Set to max speed if using 3200 MHz DIMMs. | +| AMD CBS / UMC common options / DDR4 common options / DRAM controller configuration / DRAM power options | Power down enable | Disabled | RAM power down. | +| AMD CBS / security | TSME | Disabled | Memory encryption. | + +### NBIO link clock frequency + +The NBIOs (4× per AMD EPYC™ processor) are the serializers/deserializers (SerDes) that convert and prepare I/O signals for the processor's 128 external I/O interface lanes (32 per NBIO). The NBIO link clock frequency (LCLK) controls the speed of the internal bus connecting NBIO silicon to the data fabric. All data between the processor and PCIe lanes flows through the data fabric at this frequency, so it must be forced to the maximum for optimal PCIe performance. + +For AMD EPYC™ 7003-series processors, configuring all NBIOs to **Enhanced Preferred I/O** mode (see BIOS table above) is sufficient to enable the highest LCLK frequency. The AMD-IOPM-UTIL utility required for EPYC 7002 (MI100) is not needed on EPYC 7003 systems. + +### NPS memory configuration + +For the number of NUMA nodes per socket (NPS), follow the guidance of the [HPC Tuning Guide for AMD EPYC™ 7003 Series Processors](https://docs.amd.com/v/u/en-US/high-performance-computing-tuning-guide-amd-epyc7003-series-processors) for the optimal host configuration. For most HPC workloads, NPS4 is the recommended value. + ## Acceptance criteria The MI210 system acceptance process validates that the platform is correctly configured, stable, and performing to expectations. Follow the sequence: Prerequisites → Basic Health Checks → System Validation → Performance Benchmarks. @@ -72,7 +122,7 @@ Ensure all system requirements are met before proceeding with validation. See th - ✅ Supported operating system version installed - ✅ Compatible ROCm version installed -- ✅ BIOS configured per [BIOS settings](../common/bios-settings.md), with MI210-specific values per platform vendor +- ✅ BIOS configured per [MI210 BIOS settings](#bios-settings) (EPYC 7003-series baseline; verify power limits with platform vendor) - ✅ Required kernel parameters present: `pci=realloc=off`, `pci=bfsort`, `iommu=pt`, and `amd_iommu=on` (or `intel_iommu=on` on Intel hosts) — see [Kernel Parameters](../common/kernel-parameters.md) - ✅ Minimum 512G system memory available - ✅ Latest applicable firmware applied consistently across nodes diff --git a/docs/gpus/mi250.md b/docs/gpus/mi250.md index a1bce0d..a8408fc 100644 --- a/docs/gpus/mi250.md +++ b/docs/gpus/mi250.md @@ -34,7 +34,7 @@ For the most up-to-date information on supported operating systems and distribut [ROCm docs](https://rocm.docs.amd.com) is the single source of truth for supported versions, distribution compatibility, and required dependencies for the ROCm toolkit. ``` -For BIOS, NUMA, and OS-level tuning that applies to all AMD Instinct hosts, see [BIOS settings](../common/bios-settings.md) and [OS tuning](../common/os-tuning.md). +For general NUMA and OS-level tuning that applies to all AMD Instinct hosts, see [OS tuning](../common/os-tuning.md). MI250-specific BIOS settings are in the [BIOS settings](#bios-settings) section below. ### GPU identification @@ -59,6 +59,57 @@ Expected output example: The 8 GCDs are paired by OAM (e.g. `11`+`14` are the two GCDs on one OAM, `32`+`35` on the next, and so on). Same-OAM GCD pairs are connected by a high-bandwidth on-package link, while cross-OAM connectivity uses external xGMI ports in a 4-OAM all-to-all mesh. +## BIOS settings + +For maximum MI250 GPU performance on systems with AMD EPYC™ 7003-series processors (codename "Milan") and AMI System BIOS, the following BIOS settings have been validated. These settings should be set as default values in the system BIOS. Analogous settings for other non-AMI BIOS providers can be configured similarly. On systems with Intel processors, some settings may not apply. + +```{note} +The BIOS setting locations and names may vary by hardware and BIOS vendor. Consult your system documentation for details. +``` + +| BIOS setting location | Parameter | Value | Comments | +|---|---|---|---| +| Advanced / PCI subsystem settings | Above 4G decoding | Enabled | GPU large BAR support. | +| Advanced / PCI subsystem settings | SR-IOV support | Disabled | Disable single root IO virtualization. | +| AMD CBS / CPU common options | Global C-state control | Auto | Global C-states. | +| AMD CBS / CPU common options | CCD/Core/Thread enablement | Accept | May be necessary to enable the SMT control menu. | +| AMD CBS / CPU common options / performance | SMT control | Disable | Set to Auto if the primary application is not compute-bound. | +| AMD CBS / DF common options / memory addressing | NUMA nodes per socket | NPS 1, 2, or 4 | See [NPS memory configuration](#nps-memory-configuration) below. | +| AMD CBS / DF common options / memory addressing | Memory interleaving | Auto | Depends on NPS setting. | +| AMD CBS / DF common options / link | 4-link xGMI max speed | 18 Gbps | Set to highest rate supported by the CPU. | +| AMD CBS / NBIO common options | IOMMU | Disabled | | +| AMD CBS / NBIO common options | PCIe ten bit tag support | Auto | | +| AMD CBS / NBIO common options | Preferred IO | Bus | | +| AMD CBS / NBIO common options | Preferred IO bus | Use `lspci` to find PCIe device ID | | +| AMD CBS / NBIO common options | Enhanced Preferred IO mode | Enabled | | +| AMD CBS / NBIO common options / SMU common options | Determinism control | Manual | | +| AMD CBS / NBIO common options / SMU common options | Determinism slider | Power | | +| AMD CBS / NBIO common options / SMU common options | cTDP control | Manual | Set cTDP to the maximum supported by the installed CPU. | +| AMD CBS / NBIO common options / SMU common options | cTDP | 280 | Value in watts. | +| AMD CBS / NBIO common options / SMU common options | Package power limit control | Manual | Set to the maximum supported by the installed CPU. | +| AMD CBS / NBIO common options / SMU common options | Package power limit | 280 | Value in watts. | +| AMD CBS / NBIO common options / SMU common options | xGMI link width control | Manual | Set AMD CPU xGMI width to 16 bits. | +| AMD CBS / NBIO common options / SMU common options | xGMI force link width | 2 | 0: x2 / 1: x8 / 2: x16 | +| AMD CBS / NBIO common options / SMU common options | xGMI force link width control | Force | | +| AMD CBS / NBIO common options / SMU common options | APBDIS | 1 | Disable DF P-states. | +| AMD CBS / NBIO common options / SMU common options | DF C-states | Enabled | | +| AMD CBS / NBIO common options / SMU common options | Fixed SOC P-state | P0 | | +| AMD CBS / UMC common options / DDR4 common options | Enforce POR | Accept | | +| AMD CBS / UMC common options / DDR4 common options / Enforce POR | Overclock | Enabled | | +| AMD CBS / UMC common options / DDR4 common options / Enforce POR | Memory clock speed | 1600 MHz | Set to max speed if using 3200 MHz DIMMs. | +| AMD CBS / UMC common options / DDR4 common options / DRAM controller configuration / DRAM power options | Power down enable | Disabled | RAM power down. | +| AMD CBS / security | TSME | Disabled | Memory encryption. | + +### NBIO link clock frequency + +The NBIOs (4× per AMD EPYC™ processor) are the serializers/deserializers (SerDes) that convert and prepare I/O signals for the processor's 128 external I/O interface lanes (32 per NBIO). The NBIO link clock frequency (LCLK) controls the speed of the internal bus connecting NBIO silicon to the data fabric. All data between the processor and PCIe lanes flows through the data fabric at this frequency, so it must be forced to the maximum for optimal PCIe performance. + +For AMD EPYC™ 7003-series processors, configuring all NBIOs to **Enhanced Preferred I/O** mode (see BIOS table above) is sufficient to enable the highest LCLK frequency. The AMD-IOPM-UTIL utility required for EPYC 7002 (MI100) is not needed on EPYC 7003 systems. + +### NPS memory configuration + +For the number of NUMA nodes per socket (NPS), follow the guidance of the [HPC Tuning Guide for AMD EPYC™ 7003 Series Processors](https://docs.amd.com/v/u/en-US/high-performance-computing-tuning-guide-amd-epyc7003-series-processors) for the optimal host configuration. For most HPC workloads, NPS4 is the recommended value. + ## Acceptance criteria The MI250 system acceptance process validates that the platform is correctly configured, stable, and performing to expectations. Follow the sequence: Prerequisites → Basic Health Checks → System Validation → Performance Benchmarks. @@ -78,7 +129,7 @@ Ensure all system requirements are met before proceeding with validation. See th - ✅ Supported operating system version installed - ✅ Compatible ROCm version installed (verify: `cat /opt/rocm/.info/version`); see the [ROCm System Requirements](https://rocm.docs.amd.com/projects/install-on-linux/en/latest/reference/system-requirements.html) for the current supported version matrix -- ✅ BIOS configured per [BIOS settings](../common/bios-settings.md), with MI250-specific values per platform vendor +- ✅ BIOS configured per [MI250 BIOS settings](#bios-settings) (EPYC 7003-specific table above) - ✅ Required kernel parameters present: `pci=realloc=off iommu=pt` - ✅ Minimum 1T system memory available - ✅ Latest applicable firmware applied consistently across nodes diff --git a/docs/gpus/mi300a.md b/docs/gpus/mi300a.md index bf24b10..b4eb82b 100644 --- a/docs/gpus/mi300a.md +++ b/docs/gpus/mi300a.md @@ -48,6 +48,89 @@ Expected output example: 0003:01:00.0 Processing accelerators: Advanced Micro Devices, Inc. [AMD/ATI] Aqua Vanjaram [Instinct MI300A] ``` +## APU-specific OS tuning + +Because the MI300A integrates CPU and GPU on a single package with shared HBM3, several OS-level settings that are optional on discrete OAM systems are required or strongly recommended here. + +### Transparent huge pages + +The `transparent_hugepage=always` kernel parameter is required on MI300A. The unified CPU+GPU shared-memory model depends on transparent huge pages for correct and performant operation. Add it to `GRUB_CMDLINE_LINUX` and update GRUB as described in [Kernel Parameters](../common/kernel-parameters.md). + +To verify the setting is active after boot: + +```bash +cat /sys/kernel/mm/transparent_hugepage/enabled +``` + +The output should show `[always]` as the active selection. + +### Increase allocatable GPU memory (`amdttm`) + +By default, only 96 GiB of the 128 GiB HBM per APU is allocatable via HIP device allocations. To make the full 128 GiB available, load the `amdttm` module with updated page limits. The values correspond to the number of 4 KiB pages. + +Check the current limit: + +```bash +cat /sys/module/amdttm/parameters/pages_limit +``` + +To set the full 128 GiB across all four APUs on a single-node system (total 512 GiB), run: + +```bash +sudo modprobe amdttm pages_limit=134217728 page_pool_size=134217728 +sudo modprobe amdgpu +``` + +```{note} +For a two-socket node with two APUs, halve the values: `pages_limit=67108864 page_pool_size=67108864`. +The per-APU limit is `33554432` pages × number of APUs on the node. +``` + +To make the setting persistent, add it to `/etc/modprobe.d/amdttm.conf`: + +```text +options amdttm pages_limit=134217728 +options amdttm page_pool_size=134217728 +``` + +If the filesystem is not yet mounted when the kernel driver loads (for example, on early-boot initrd), pass the values as boot parameters instead: + +```text +amdttm.pages_limit=134217728 amdttm.page_pool_size=134217728 +``` + +Verify the new setting: + +```bash +cat /sys/module/amdttm/parameters/pages_limit +``` + +### Limit GPU memory allocation percentage + +Many AI applications were originally developed for discrete GPUs and probe available memory until exhaustion. On the MI300A APU, where CPU and GPU share the same HBM pool, this can starve the OS of memory. + +Use the following environment variables to bound how much of the shared HBM the GPU side can claim: + +```bash +export GPU_MAX_ALLOC_PERCENT= +export GPU_SINGLE_ALLOC_PERCENT= +``` + +Setting either variable to `100` allows the GPU to allocate all free memory. Before doing so, account for expected CPU workload and OS usage. For example, on a 128 GiB system where the OS requires 8 GiB, setting `GPU_MAX_ALLOC_PERCENT=100` authorizes a single workload to allocate up to 120 GiB. Any over-allocation is handled by the kernel's out-of-memory (OOM) policies unless swap space is configured. + +Add these variables to `/etc/environment` or the user shell environment. + +### Enable memory compaction + +The MI300A APU dynamically shares memory between the CPU and GPU. Without periodic memory compaction, fragmentation accumulates and application performance degrades over time. Compaction is enabled by default on Ubuntu but disabled by default on RHEL-based distributions. + +Enable proactive compaction: + +```bash +echo 20 > /proc/sys/vm/compaction_proactiveness +echo 1 > /proc/sys/vm/compact_unevictable_allowed +``` + ## Acceptance criteria The MI300A system acceptance process validates that the platform is correctly configured, stable, and performing to expectations. Follow the sequence: Prerequisites → Basic Health Checks → System Validation → Performance Benchmarks. @@ -69,10 +152,11 @@ Ensure all system requirements are met before proceeding with validation. See th - ✅ Compatible ROCm version installed (verify: `cat /opt/rocm/.info/version`); see the [ROCm System Requirements](https://rocm.docs.amd.com/projects/install-on-linux/en/latest/reference/system-requirements.html) for the current supported version matrix - ✅ BIOS configured per [BIOS settings](../common/bios-settings.md), with MI300A-specific values per platform vendor (IOMMU off, memory interleaving, NPS) - ✅ Required kernel parameters present: `pci=realloc=off transparent_hugepage=always numa_balancing=disable` -- ✅ Sysctl tunings applied: `vm.compaction_proactiveness=20`, `vm.max_map_count` increased per ROCm guide +- ✅ Sysctl tunings applied: `vm.compaction_proactiveness=20` and `vm.compact_unevictable_allowed=1` (required on RHEL-based systems; enabled by default on Ubuntu) — see [Enable memory compaction](#enable-memory-compaction) +- ✅ `amdttm` page pool configured for full 128 GiB HBM access per APU — see [Increase allocatable GPU memory](#increase-allocatable-gpu-memory-amdttm) - ✅ Environment variables (where applicable): - `HSA_OVERRIDE_CPU_AFFINITY_DEBUG=0` - - `GPU_MAX_ALLOC_PERCENT` and `GPU_SINGLE_ALLOC_PERCENT` tuned per workload + - `GPU_MAX_ALLOC_PERCENT` and `GPU_SINGLE_ALLOC_PERCENT` tuned per workload — see [Limit GPU memory allocation percentage](#limit-gpu-memory-allocation-percentage) - ✅ Minimum 4 × 128 GB = 512 GB unified HBM3 visible to the OS usable host-visible memory (note: MI300A's HBM is unified with CPU) - ✅ Latest applicable firmware applied consistently across nodes - ✅ ROCm Validation Suite (RVS) installed @@ -83,8 +167,9 @@ These checks ensure fundamental system health and proper APU detection. For deta | Test | Command | Pass/Fail criteria | |------|---------|-------------------| -| [Check OS distribution](../common/health-checks.md#check-os-distribution) | `cat /etc/os-release` | **Pass**: OS version listed in compatibility matrix
**Fail**: Otherwise | +| [Check OS distribution](../common/health-checks.md#check-os-distribution) | `cat /etc/os-release` | **Pass**: OS version listed in compatibility matrix; kernel ≥ 5.18
**Fail**: Otherwise | | [Check kernel boot arguments](../common/health-checks.md#check-kernel-boot-arguments) | `cat /proc/cmdline` | **Pass**: Contains `pci=realloc=off transparent_hugepage=always numa_balancing=disable`
**Fail**: Missing any required param | +| Verify IOMMU is off (MI300A-specific) | `sudo apt install acpica-tools && sudo acpidump \| grep IVRS && sudo acpidump \| grep DMAR` | **Pass**: Both commands return no output (IOMMU disabled)
**Fail**: Either command returns output | | [Check for driver errors](../common/health-checks.md#check-for-driver-errors) | `sudo dmesg -T \| grep amdgpu \| grep -i error` | **Pass**: Null
**Fail**: Errors reported | | [Check available memory](../common/health-checks.md#check-for-available-system-memory) | `lsmem \| grep "Total online memory"` | **Pass**: ≥ 4 × 128 GB = 512 GB unified HBM3 visible to the OS
**Fail**: Less than 4 × 128 GB = 512 GB unified HBM3 visible to the OS | | [Check GPU presence](../common/health-checks.md#check-gpu-presence) | `sudo lspci -d 1002:74a0` | **Pass**: 4 MI300A APUs found
**Fail**: Otherwise |