cmk-amd-mi355x: validation suite for MI355X + Pollara 400 on CMK

[rkalmankar-crusoe]

What this adds

A reproducible validation bundle for AMD Instinct MI355X nodes (8× GPU, 8× Pensando
Pollara 400 @ 400 Gbps) on Crusoe Managed Kubernetes, pinned to Bundle B.MI355.2.1
(ROCm 7.2.0, RCCL 2.27.7, AINIC fw 1.117.5-a-77). Each check is a self-contained
folder with the runnable script or manifest, targeting a live 2-node cluster.

Checks

• env-verify — per-node kernel / driver / GPU / NIC / dmesg audit against the Bundle 2.1 spec
• rccl-allreduce — 2-node RCCL all_reduce_perf MPIJob with the Crusoe + mlcommons-tuned envelope
• gpu-straggler-scan — per-GPU compute burst, sustained GEMM, XGMI mesh, ECC + bad-pages
• gpu-to-nic-bw-hostmem / -gpu-direct — per-rail bidirectional ib_write_bw, host-memory baseline and GPU-direct dma-buf (--use_rocm --use_rocm_dmabuf)
• build-image / mirror-image — in-cluster Kaniko build of the AMD-patched perftest image, and a one-time skopeo mirror of the Bundle base image

Reference numbers (internal dry-run, 2 × MI355X-288GB-ROCE.8x)

• RCCL all-reduce: 381.88 GB/s avg bus BW (95.5% of theoretical peak)
• Per-rail RDMA: ~778 Gb/s bidirectional on all 16 rails, 0.2% spread — identical for
  host-memory and GPU-direct dma-buf, proving RDMA-into-HBM is live end-to-end
• Compute: 16/16 GPUs healthy, sustained bf16 GEMM 1331–1399 TFLOPS (92–97% of vendor spec)

These set the acceptance bar for the upcoming customer POC on the same platform.

Notes for reviewers

• README documents run order, expected outputs, and interpretation per check
• logs/ and results/ are runtime-created and gitignored
• GPU-direct check requires the build-image step first (AMD-patched perftest)