BF16 (BFloat16)

BF16 (Brain Floating Point 16) is a 16-bit floating-point number format that uses 8 exponent bits and 7 mantissa bits, matching the dynamic range of FP32 while halving memory usage. In local AI, BF16 is used for model inference and training on compatible hardware (e.g., NVIDIA Ampere+ GPUs, Apple M-series). It reduces VRAM consumption by roughly half compared to FP32, enabling larger models or longer contexts to fit on consumer GPUs. However, not all hardware supports BF16 natively; older GPUs may fall back to slower FP16 emulation.

BF16 was developed by Google Brain for deep learning, prioritizing dynamic range over precision. Its 8-bit exponent matches FP32's exponent range, preventing overflow/underflow issues common in FP16, while the reduced mantissa precision (7 bits vs FP16's 10) is acceptable for gradient updates and inference. In practice, BF16 offers a sweet spot: it uses half the memory of FP32, runs efficiently on tensor cores (e.g., NVIDIA's Ampere architecture and newer), and avoids the numerical instability of FP16 for many models. Operators see BF16 as a default precision in frameworks like PyTorch (via torch.bfloat16) and in inference engines like vLLM. On Apple Silicon, BF16 is supported via MLX and Metal Performance Shaders, though FP16 is more common. For local inference, BF16 is often used in quantized models (e.g., Q4_K_M) but as a base precision for weights before quantization. The main trade-off: BF16 requires hardware support for full speed; on unsupported GPUs, it may be emulated in software, reducing performance.

An operator with an RTX 4090 (24 GB VRAM) loads Llama 3.1 70B in BF16: the model requires ~140 GB in FP32, but BF16 halves that to ~70 GB — still too large. So they use 4-bit quantization (Q4_K_M) which compresses further to ~40 GB, fitting with offloading. If they had an RTX 3090 (24 GB) without BF16 tensor cores, BF16 would run slower due to emulation. For a 7B model, BF16 uses ~14 GB, fitting comfortably on a 16 GB GPU.

In llama.cpp, when compiling with LLAMA_CUDA=1, the runtime checks for BF16 support via CUDA compute capability (8.0+). Operators can force BF16 by setting environment variable GGML_CUDA_FORCE_BF16=1. In Ollama, pulling a model like llama3.1:8b downloads weights in Q4_K_M by default, but if using --precision bf16, the runtime attempts BF16. In Hugging Face Transformers, loading with model = AutoModelForCausalLM.from_pretrained("meta-llama/Llama-2-7b-hf", torch_dtype=torch.bfloat16) uses BF16 if the GPU supports it.