AMD Radeon RX 6900 XT

This card is for the operator who wants to run 13B and 32B models at home without compromise, and who is comfortable with AMD's ROCm stack. The RX 6900 XT delivers ~95-130 tok/s on 7B Q4, ~50-70 tok/s on 13B Q4, and can handle 32B Q4 with partial offload at ~20-30 tok/s. The 16 GB VRAM is a sweet spot for local inference, enabling full 13B Q4 models and most 32B Q4 models with some offloading. What breaks: ROCm support, while official, still lags behind CUDA in ecosystem maturity—some newer model architectures or custom kernels may require workarounds. The 300W TDP means the rig needs adequate cooling and power supply. When to pass: if the workload demands 70B+ models or requires plug-and-play compatibility with the latest PyTorch features, a used RTX 3090 with 24 GB VRAM and CUDA is a better fit. At ~$500 used, this card offers strong value for local AI inference, especially for operators willing to navigate ROCm's quirks.