ROCm

What ROCm actually is

ROCm is AMD's open-source GPU compute stack — the equivalent layer to NVIDIA's CUDA. It includes the HIP programming model (a C++ runtime that ports cleanly between AMD and NVIDIA at the source level), HIPify tooling that auto-translates CUDA source to HIP, and accelerated math libraries (rocBLAS, rocFFT, MIOpen) that AMD GPUs need to compete with cuDNN-class performance on AI workloads.

vLLM, SGLang and PyTorch on AMD GPUs run on ROCm, as does the HIP build of llama.cpp. llama.cpp and Ollama also offer a Vulkan path that does not use ROCm.

Where it fits in the stack

ROCm is a driver / runtime layer, not an inference engine. The stack:

• Hardware: AMD GPU — RDNA3 (RX 7900 series) or Instinct (MI300, MI250) is the realistic 2026 floor for serious AI work
• Driver / runtime: AMD's amdgpu kernel driver + ROCm userspace
• Compute libraries: rocBLAS, MIOpen, hipBLASLt, rocSPARSE
• Inference engines: llama.cpp, PyTorch (with ROCm wheels), vLLM, SGLang, ExLlamaV2 (limited), bitsandbytes (limited)
• Frontends: Ollama, LM Studio, Open WebUI

Picking AMD for local AI in 2026 means picking the ROCm path for vLLM, SGLang and PyTorch; llama.cpp and Ollama also offer Vulkan.

Best use cases

• High-VRAM inference workstations. An RX 7900 XTX gives 24 GB VRAM.
• Datacenter MI300X / MI250X clusters. Where AMD has invested most heavily; ROCm 6+ is competitive with CUDA on H100-class workloads at the kernel level for the workloads AMD has tuned.
• PyTorch researchers with AMD hardware. ROCm PyTorch wheels are a one-line install; most research code runs unchanged.

OS support

For Windows AMD users, the practical truth in May 2026 is: ROCm on Windows works for llama.cpp and Ollama for most inference cases, but breaks under more advanced workloads (multi-GPU tensor-parallel, some PyTorch ops, FlashAttention variants). If you can dual-boot Linux, do it.

Hardware support

ROCm officially supports a narrower hardware list than CUDA. The 2026 working list:

• Datacenter: MI355X, MI350X, MI350P, MI325X, MI300X, MI300A, MI250X, MI250, MI210, MI100
• Consumer (RDNA4): RX 9070 XT, RX 9070 GRE, RX 9070, RX 9060 XT, RX 9060, RX 9050
• Consumer (RDNA3): RX 7900 XTX, RX 7900 XT, RX 7900 GRE, RX 7800 XT, RX 7700 XT, RX 7700, RX 7600
• APUs: Ryzen AI Max 300, Ryzen AI Max PRO 400, Ryzen AI 300 and Ryzen AI 400 series, among others
• Consumer (RDNA2): RX 6900 XT, RX 6800 XT (community-supported; gfx1030 target)

If you are buying new AMD for local AI in 2026, RDNA3 is the floor; older cards work but the Linux toolchain quality drops sharply below RDNA2.

Model / inference engine compatibility

This is where the rubber meets the road, and the picture has improved a lot but still has gaps:

• llama.cpp — full GGUF support via its HIP backend; the most reliable AMD inference path in 2026
• PyTorch (ROCm wheels) — installs in one line; most research code runs unchanged
• vLLM (ROCm) — supported on MI300X / MI250X; consumer RDNA3 support exists but is rougher
• SGLang — partial AMD support; lags vLLM
• ExLlamaV2 — limited AMD support; the EXL2 quant kernels are CUDA-tuned
• bitsandbytes — partial; AMD support has been the long pole for a year+
• TensorRT-LLM — NVIDIA-only

Quant formats: GGUF works everywhere; AWQ / GPTQ work on PyTorch + ROCm but slower than NVIDIA equivalents; FP8 / EXL2 are NVIDIA-territory.

Setup path

On Ubuntu 24.04, the quick start in AMD's ROCm 10.1 install guide is an interactive installer:

curl -fsSLO https://stable.repo.amd.com/rocm/core/linux-installers/rocm-installer-10.1.0-5.run && bash rocm-installer-10.1.0-5.run
sudo usermod -a -G render,video $LOGNAME
# reboot to apply all settings
rocminfo                    # should list your GPU

Then for PyTorch:

pip3 install torch torchvision --index-url https://download.pytorch.org/whl/rocm10.0

For llama.cpp:

HIPCXX="$(hipconfig -l)/clang" HIP_PATH="$(hipconfig -R)" \
    cmake -S . -B build -DGGML_HIP=ON -DGPU_TARGETS=gfx1100 -DCMAKE_BUILD_TYPE=Release   # 7900 XTX
cmake --build build --config Release -j

What breaks first

1. Wrong gfx target compiled in. ROCm binaries are GPU-arch-specific. Building for gfx1100 (7900 XTX) and trying to load on gfx1030 (6900 XT) silently fails. See /errors/rocm-device-not-found.
2. Driver / ROCm version mismatch. amdgpu kernel driver and ROCm userspace must agree. Distro upgrades break this.
3. Out-of-tree kernel. Custom kernels (Zen4-tuned, TKG, etc.) need the amdgpu-dkms package re-built; usually fails silently.
4. vLLM / SGLang version drift. ROCm support tracks behind CUDA support by 1-2 minor versions; pin everything.
5. Multi-GPU tensor-parallel. Works on MI300X clusters; consumer RDNA3 multi-GPU TP is still flaky in 2026.

Alternatives by intent

Best pairings

• RX 7900 XTX + ROCm + llama.cpp = a 24 GB VRAM local inference path
• MI300X cluster + ROCm + vLLM = the AMD datacenter answer to H100 + TensorRT-LLM
• Ubuntu 24.04 LTS — the reference OS; everything else is harder

Who should avoid ROCm

• Time-constrained operators. ROCm has caught up enormously but still requires more debugging time than CUDA. If you are paid by the hour to ship local AI, NVIDIA is cheaper.
• Anyone whose stack depends on bleeding-edge NVIDIA-tuned kernels (FP8 transformer engine, latest FlashAttention variants, EXL2).
• macOS users. Use MLX-LM instead.

Related

• Hardware: RX 7900 XTX
• Tools: llama.cpp, vLLM, Ollama
• System guides: /setup, /compatibility
• Errors: /errors/rocm-device-not-found, /errors/wsl2-gpu-not-detected