Vulkan Compute

Vulkan compute is the cross-vendor GPU compute API from Khronos. llama.cpp ships a Vulkan backend that runs on AMD, Intel, and NVIDIA GPUs without vendor-specific drivers — making it the most portable GPU path for local inference.

Performance is typically 70–90% of the vendor-native path (CUDA on NVIDIA, ROCm on AMD). The win is portability: if your GPU is too old for ROCm or you're on an AMD APU, Vulkan is often the only path that works.

Limitations: no FP16 storage on some Intel iGPUs, no support for multi-GPU split, some quants (older K-quants) not yet implemented in the Vulkan kernels.

An operator inherits a mixed fleet — a couple of older Radeon RX 580s without ROCm support and an Intel Arc A770 — and needs one inference path that works on all of them without maintaining separate CUDA/ROCm builds. Compiling llama.cpp with the Vulkan compute backend (-DGGML_VULKAN=ON) gets a 7B Q4_K_M model running on every card with a single binary, no vendor SDK installs beyond the Vulkan loader. Throughput lands noticeably below what the A770 would hit with a mature vendor-native backend, but for a fleet where half the hardware has no first-class vendor path at all, "workable everywhere" beats "fastest on one card." The team keeps a CUDA build for their single RTX-equipped node and Vulkan for everything else, rather than fighting ROCm's hardware support matrix on the old Radeons.