ROCm (AMD)

ROCm (Radeon Open Compute) is AMD's open-source equivalent of NVIDIA's CUDA. It's required for any meaningful AMD GPU inference — vLLM ROCm builds, llama.cpp HIP backend, ExLlamaV2 ROCm, PyTorch ROCm. Without it, AMD cards fall back to CPU or Vulkan, which is dramatically slower for LLM inference.

In 2026, ROCm is mature on Linux for current-generation consumer cards (RX 7900 XTX, RX 9070 XT) and datacenter chips (MI300, MI250). Older Polaris and Vega cards are unsupported in current ROCm — confirm your card is in the support matrix before committing time. Windows ROCm is improving but trails Linux by 6-12 months for LLM workloads; production AMD deployments live on Linux.

The operator-honest framing: AMD-on-Linux is a real production path; AMD-on-Windows is hobby-tier in 2026. ROCm 6.x supports the same patterns as CUDA — Docker containers via amdgpu-container-toolkit, Triton kernels via HIPify, FA2/3 ports — but the community + tooling density still trails CUDA. Choose AMD when budget is the constraint and your team can run Linux; choose NVIDIA when ecosystem maturity matters more than card price.

An operator on a budget builds a local inference box around an RX 7900 XTX (24 GB, roughly half the price of an RTX 4090 with comparable VRAM) and needs it to actually beat CPU-only inference. Installing ROCm on Ubuntu and running llama.cpp's HIP backend gets them GPU-accelerated decode competitive with NVIDIA cards of similar VRAM class. The same card under Windows, using DirectML or Vulkan fallback instead of a mature ROCm build, runs noticeably slower and lags behind current llama.cpp features. Before buying, they check AMD's ROCm support matrix — an older Vega-class card would silently fall back to CPU-speed inference, wasting the purchase. Their conclusion: AMD-on-Linux is the honest budget path to more VRAM per dollar; AMD-on-Windows is still a hobbyist setup in 2026.