Llama

Llama 1 (65B/33B/13B/7B) debuted in February 2023 as a research-only release — dense transformer decoder with pre-norm RMSNorm, SwiGLU FFN, and rotary position embeddings (RoPE). The architecture established three principles that carried forward: decoder-only dense, grouped-query attention for throughput, and vocab size of 32K tokens trained on publicly available data. Llama 1's 65B matched GPT-3.5 on several benchmarks despite having ~3.5× fewer parameters, validating the Chinchilla-optimal compute scaling approach Meta followed. Llama 2 (July 2023) added doubled context length (4K tokens), GQA expanding from 32K-head to 40K-vocab model, and a more aggressive data mix (2T tokens vs 1.4T). The commercial-friendly Llama 2 Community License transformed the open-weight landscape overnight — enterprises could now deploy Llama 2 in production without the research-only constraints of Llama 1. The architectural changes were modest: same RoPE, same SwiGLU, same pre-norm. The value proposition was license + data scale, not architectural novelty. Training on 15T tokens, Llama 3 405B was the first open-weight model competitive with GPT-4 class. The architecture kept the dense transformer recipe but scaled aggressively: grouped-query attention ratio of 8 key-value heads, 128 layers for the 405B variant, and training on 16K H100 GPUs. Anecdotal reports suggest 54 days of training at 400 TFLOPS/GPU sustained. Llama 3.1 (July 2024) extended context to 128K tokens via RoPE interpolation and added native multilingual support across 8 languages beyond English. The 405B model received minor RLHF improvements. [Llama 3.3](/families/llama) (December 2024) refined the 70B model specifically — the 70B variant of 3.3 closed the gap to the legacy 405B on reasoning benchmarks while being far more deployable on single-node hardware. This is the currently most-referenced architectural state. [Llama 4](/families/llama) (Behemoth/Maverick/Scout, 2026) introduces the first architectural divergence: MoE for the Scout variant and continued dense scaling for Behemoth. Early reports suggest Llama 4 Scout uses 16 experts with top-2 routing at ~1T total parameters and ~50B active per token. RoPE continues as the positional encoding backbone. The architectural evolution from Llama 1→4 traces a clear path: dense decoder → dense decoder at frontier scale → MoE diversification while preserving the Llama inference API contract.

Llama 1 (all sizes) ran on consumer hardware at release — the 65B at Q4 could fit on 2× [RTX 3090](/hardware/rtx-3090) or a single [RTX A6000](/hardware/nvidia-a40) with 48 GB VRAM. The 13B ran comfortably on a single [RTX 3060 12GB](/hardware/rtx-3060-12gb) at Q4. This was the singular value proposition of Llama 1: frontier-capability at consumer-accessible compute. Llama 2 preserved this accessibility. The 70B at Q4 (~40 GB) required 2× 24 GB cards or an [RTX A6000 48GB](/hardware/nvidia-a40). The 13B at Q4 (~8 GB) ran comfortably on a single [RTX 3060 12GB](/hardware/rtx-3060-12gb). The commercial license meant production deployment on consumer GPUs was viable for the first time — small businesses ran fine-tuned Llama 2 13B on a single [RTX 4090](/hardware/rtx-4090) for internal tooling. Llama 3 shifted the accessibility curve dramatically. The 8B at Q4 (~5 GB) runs on a [MacBook Pro M4 Max](/hardware/macbook-pro-16-m4-max) easily. The 70B at Q4 (~40 GB) still fits 2× [RTX 4090](/hardware/rtx-4090). But the 405B variant is not local-friendly: FP16 requires ~810 GB of memory. Q4 (~230 GB) requires [Mac Studio M3 Ultra](/hardware/mac-studio-m3-ultra) at 192 GB with aggressive compression or 3× [RTX 5090](/hardware/rtx-5090). For context: a single [H100 SXM](/hardware/nvidia-h100-sxm) at 80 GB cannot fit Llama 3 405B even at Q4 without tensor parallelism across at least 3 cards. The trajectory is bifurcated: small models (8B class) got more deployable over time (better quant, [MLX-LM](/tools/mlx-lm) on Apple Silicon, [Ollama](/tools/ollama) one-click), while large models (405B class) are increasingly datacenter-only. Llama 3.3 70B represents the practical ceiling for single-node local deployment in 2026 — it fits on [Mac Studio M3 Ultra](/hardware/mac-studio-m3-ultra) at Q4 using 40 GB, on [RTX 5090](/hardware/rtx-5090) 32 GB at IQ3-XS, or comfortably across 2× [RTX 4090](/hardware/rtx-4090) at Q4 without tensor parallelism. Llama 4 Behemoth is functionally a datacenter-only model. The takeaway: the Llama family's local viability bifurcated around the Llama 3 generation. Sub-70B models remain the local sweet spot. For readers choosing a Llama model for local inference: if you have < 24 GB VRAM, stick with Llama 3.1 8B at Q4 or Q5. If you have 48-64 GB (2× 24 GB or single [Apple M3 Ultra](/hardware/apple-m3-ultra)), Llama 3.3 70B at Q4 is the practical ceiling.

Llama 1 65B scored 63.4% on 5-shot MMLU at release — competitive with GPT-3.5 (70.0%) at roughly one-third the parameters. On GSM8K 8-shot, 50.9% was adequate but not exceptional. HellaSwag 76.2% and ARC-Challenge 57.8% rounded out the picture: a capable base model outperforming every open-weight model of its era but trailing GPT-3.5-turbo on reasoning-heavy benchmarks. Llama 2 70B improved MMLU to 68.9% (5-shot) and GSM8K to 56.8% (8-shot). The gains were meaningful but incremental — the data scaling had diminishing returns on reasoning benchmarks. HumanEval pass@1 for the 70B Instruct variant reached 29.9%, which was serviceable for code generation but not class-leading. The standout metric for Llama 2 was truthfulness: TruthfulQA MC2 of 65.1% for the 70B, better than GPT-3.5's 60.5%, suggesting the RLHF recipe was effective at hallucination reduction. Llama 3 405B was the inflection point. MMLU 5-shot reached 85.2%, GSM8K 8-shot 89.0%, and HumanEval pass@1 54.9%. IFEval strict-prompt accuracy hit 77.6%, demonstrating strong instruction-following at scale. The 70B variant scored MMLU 79.5% — closing within 6 points of the 405B flagship while using 5.8× fewer parameters. This parameter-efficiency curve was the strongest signal that Llama 3 training data quality was the differentiator, not just scale. Llama 3.1 405B pushed MMLU to 87.3%, GSM8K to 91.2%, and HumanEval to 61.0%. The 128K context evaluation on RULER showed 94.3% retrieval accuracy at 128K tokens, competitive with GPT-4-128K. Multilingual MMLU (Arabic, German, French, Hindi, Italian, Portuguese, Spanish, Thai) averaged comparable performance to English MMLU, a first for the Llama family. This was Llama's "GPT-4 parity" generation on benchmarks. Llama 3.3 70B matched or exceeded Llama 3.1 405B on reasoning benchmarks: MMLU 86.9%, GSM8K 90.5%. The 70B post-training improvements via iterative DPO and better reward modeling had closed the quality gap. On HumanEval, the 70B reached 58.5% — slightly behind the 405B's 61.0% but at ~6× lower serving cost where the family wins its value proposition. The family's benchmark trajectory follows a clear pattern: each generation roughly matches the larger variant of the previous generation at 5-6× lower per-token serving cost. Where Llama wins versus [Qwen](/families/qwen): broader language coverage and higher truthfulness benchmarks. Against [Mistral](/families/mistral): Llama wins at the 70B+ tier; Mistral wins at the sub-25B tier on coding.

Llama models receive first-class support from virtually every inference engine on day zero of each release. The Llama architecture — dense decoder with GQA, RoPE, and SwiGLU — is the default template for inference engine development. Any engine that supports one Llama variant supports all Llama variants. Llama 3 405B on vLLM with 4× [H100 SXM](/hardware/nvidia-h100-sxm) achieves ~200 tokens/second at batch size 32 with FP8 KV-cache. For Llama 3.3 70B on a single [RTX 5090](/hardware/rtx-5090) at Q4, vLLM delivers ~90 tokens/second prefill and ~35 tokens/second decode. [SGLang](/tools/sglang) matches vLLM parity on Llama and often edges ahead on structured generation latency. SGLang's RadixAttention tree cache is particularly effective for Llama-based agent workloads where conversation prefixes are shared across parallel exploration. [Ollama](/tools/ollama) provides one-command Llama deployment: `ollama run llama3.3:70b`. Ollama's Llama support is the most heavily tested path in open-weight local inference — if Ollama breaks Llama, the internet notices. [llama.cpp](/tools/llama-cpp) is the reference implementation for Llama inference. Llama 3 70B at Q4_K_M with 4-bit KV-cache on [Apple M3 Ultra](/hardware/apple-m3-ultra) achieves ~12 tokens/second via Metal — usable for personal use. The build step is non-trivial (conversion to TensorRT engines takes ~30 min for 70B), but for sustained production, the 2-3× throughput improvement over vLLM at the same batch size justifies the upfront cost. [MLX-LM](/tools/mlx-lm) on Apple Silicon handles Llama 3.3 70B at Q4 on [Mac Studio M3 Ultra](/hardware/mac-studio-m3-ultra) via the MLX framework — ~18 tokens/second with the default 4-bit quantization, competitive with llama.cpp on Metal. MLX-LM's advantage is cleaner Python API access for building Llama-based Apple Silicon applications. [ExLlamaV2](/tools/exllamav2) targets NVIDIA GPUs with EXL2 quantization specifically optimized for Llama architectures. For single-GPU Llama 3.3 70B at 4.0 bpw on [RTX 4090 (48GB variant)](/hardware/rtx-4090), ExLlamaV2 delivers ~28 tokens/second decode with flash-attention — often the fastest single-GPU experience for Llama on NVIDIA hardware. Day-zero support on HuggingFace TGI mirrors vLLM timeline. TGI's advantage is the HuggingFace integration ecosystem — easier model-card-to-deployment pipeline than vLLM for HuggingFace-centric teams.

Llama models are the most heavily quantization-tested model family in open-weight history. Every quantization format and bit width has been benchmarked against every Llama variant. Here's what works in practice. GGUF via [llama.cpp](/tools/llama-cpp) is the reference quantization path. K-quant variants (Q2_K through Q8_0) cover the full range for each Llama model size. Llama 3.3 70B at Q4_K_M (~40 GB) loses ~0.5% MMLU accuracy vs FP16 while halving VRAM requirements. At Q3_K_M (~30 GB), the degradation is ~2% MMLU — still usable. GPTQ via AutoGPTQ targets NVIDIA GPUs with groupwise 4-bit/8-bit quantization. The calibration dataset matters: Wikitext calibration produces worse code generation quality than a code-heavy calibration mix for Llama. For production deployments, GPTQ with act-order on [RTX 4090](/hardware/rtx-4090) at 4-bit delivers the best quality-per-byte ratio. AWQ (Activation-Aware Weight Quantization) for Llama achieves ~0.2% MMLU degradation at 4-bit for the 70B variant. AWQ's advantage over GPTQ is faster quantization (~5 min vs ~30 min for 70B) and better handling of outlier channels in Llama's FFN layers. EXL2 (ExLlamaV2 format) provides mixed-precision at arbitrary bitrates (2.0-8.0 bpw). For Llama 3.3 70B at 3.5 bpw (~32 GB), the quality degradation vs FP16 is ~0.8% on MMLU and ~1.5% on GSM8K. At 4.0 bpw (~37 GB), quality is indistinguishable from GPTQ at the same bit width. FP8 via [TensorRT-LLM](/tools/tensorrt-llm) and [vLLM](/tools/vllm) is the recommended datacenter format. FP8 KV-cache quantization saves ~2× on KV-cache memory with <0.2% quality impact on long-context benchmarks (RULER 128K). This is the format for [H100 SXM](/hardware/nvidia-h100-sxm) and [B200](/hardware/nvidia-b200) deployments. FP4 via NVIDIA Blackwell hardware ([B200](/hardware/nvidia-b200)) is emerging and early results for Llama 3.3 70B are promising: ~0.8-1.2% MMLU degradation, roughly matching Q4_K_M GGUF quality but at 2× the throughput on Blackwell microarchitecture support. The key sizing rule for Llama quantization: Sub-13B models survive Q3_K_S without catastrophic degradation. 70B models survive Q3_K_M with ~2% loss — acceptable for many production workflows. 405B models need Q4 minimum for serious use; Q3_K_M on 405B Llama loses ~3-4% MMLU, which crosses the usability threshold for most production workloads.

The Llama ecosystem is the largest and most mature in open-weight AI. It is the default stack against which all other open-weight model ecosystems are measured. The ecosystem maturity is a function of three reinforcing factors: Meta's sustained investment (4 major releases in 3 years), the Llama Community License's commercial clarity (enabling production deployment), and the sheer scale of community fine-tunes (thousands of derivatives on HuggingFace). Derivative model volume is unprecedented. The Llama 3 family alone has spawned >15,000 fine-tuned variants on HuggingFace. Quantization researchers produce optimized GGUF/EXL2 versions of every major derivative within 24 hours of release. The community's self-serve infrastructure around Llama — from fine-tune recipes to quantized weights to deployment configs — is the most complete of any open-weight family, surpassing [Qwen](/families/qwen) and [Mistral](/families/mistral) in volume and speed. Corporate derivatives are significant. Databricks' Dolly 2.0 was an early Llama 1 fine-tune. Stanford's Alpaca (Llama 1) and Vicuna (Llama 1 → 2) pioneered instruction-tuning recipes. Nous Research's Capybara and Hermes series remain the most-cited Llama 3 fine-tunes. VMWare's Llama 2-powered private AI stack showed enterprise adoption patterns. License clarity is a major strength but with nuance. The Llama 3.1+ Community License permits commercial use, derivative distribution, and model modification with minimal restrictions. The primary constraint: monthly active users > 700M requires a separate license from Meta, which virtually nobody triggers. The license is significantly more commercially permissive than [Command R](/families/command-r)'s CC-BY-NC or [Gemma](/families/gemma)'s bespoke terms. This license clarity has enabled the rapid enterprise adoption of Llama-based deployments in finance, healthcare (with appropriate fine-tuning), legal tech, and developer tooling. The training data transparency is improving but remains a gap. Llama 1 was trained on "publicly available data" with no further disclosure. Llama 2 added more detail about data sources. Llama 3 disclosed 15T tokens with filtering methodology. However, Meta has not released the actual training datasets. This limits reproducibility (vs [StarCoder](/families/starcoder) which released The Stack dataset) and creates compliance uncertainty for EU AI Act documentation requirements.

Llama fine-tuning tooling is the most mature and widely documented of any open-weight family. The combination of dense architecture (making full fine-tuning feasible at 8B-70B scale), the Llama Community License (permitting fine-tune distribution), and massive community tutorial coverage has created a deep fine-tuning ecosystem. LoRA and QLoRA support is universal. HuggingFace PEFT library has first-class Llama adapter support for every generation (Llama 1 through 4). The memory sweet spot for Llama 70B QLoRA is 48 GB VRAM — any less (e.g., single [RTX 4090 24GB](/hardware/rtx-4090)) requires aggressive gradient checkpointing and reduced batch size which extends training time to >48 hours. Full fine-tuning requirements scale steeply. Llama 3 8B full fine-tune requires ~64 GB VRAM at batch size 8 with AdamW optimizer (4× parameter count for optimizer states + gradients). This means 2× [RTX 4090](/hardware/rtx-4090) or a single [A100 80GB](/hardware/nvidia-a100-80gb-sxm). Llama 3.3 70B full fine-tune requires ~560 GB VRAM — 8× [A100 80GB](/hardware/nvidia-a100-80gb-sxm) or 4× [H100 SXM](/hardware/nvidia-h100-sxm) with ZeRO-3. This is datacenter-scale. The practical advice: full fine-tune is for funded teams; LoRA/QLoRA is for everyone else. Dataset ecosystem is enormous. OpenOrca, OpenAssistant, ShareGPT, UltraChat, Dolly, Alpaca, and countless domain-specific datasets (medical, legal, code) have Llama fine-tuning recipes tested and documented. Axolotl and Unsloth are the two dominant Llama fine-tuning frameworks — both offer one-command fine-tune configs for every Llama generation with built-in dataset preprocessing, quantization-aware training, and evaluation harness. RLHF/DPO support is mature. TRL (Transformer Reinforcement Learning) library supports DPO, PPO, and reward modeling for Llama with documented examples. Llama 3 Instruct models themselves were trained with iterative DPO by Meta, establishing the recipe for community reproduction. The SPPO and KTO preference optimization methods also have Llama-tested configurations. Merging and model-soup techniques are surprisingly effective for Llama. The community has discovered that merging Llama fine-tunes from different datasets (e.g., code + chat LoRAs) produces competitive multi-task performance. MergeKit + Arcee's Mergekit are the tooling standards. The practice of "Llama fine-tune merging" is more mature for Llama than any other family except potentially [Mistral](/families/mistral).

License compliance is the most important deployment consideration for Llama. The 700M monthly active user threshold for additional licensing is irrelevant for most self-hosted deployments. If your use case involves generating training data with Llama for other models, you need to verify this doesn't trigger the license restriction. Serving performance has several known edge cases. This means a 70B 4-bit deployment at 128K context needs ~100 GB of total VRAM — not just ~40 GB for the model. The practical advice for local deployment: use context lengths ≤ 32K unless you have datacenter-grade VRAM. GGUF/llama.cpp serving has specific Llama quirks. RoPE frequency scaling must match the training configuration — Llama 3's RoPE theta of 500,000 vs Llama 2's 10,000 means context extension parameters must be version-specific. Flash attention (--flash-attn) must be explicitly enabled for Llama 3+ with long context; without it, perplexity degrades sharply above 4K tokens. Production deployment on [vLLM](/tools/[vllm](/tools/vllm)) encounters Llama-specific prefix-caching behavior. Llama's GQA makes KV-cache sharing unusually efficient for same-prefix workloads (e.g., system prompts), but this efficiency drops if users have different system prompts. For multi-tenant Llama serving, homogenizing system prompts can yield 30-40% throughput improvement from prefix cache hits. Safety and alignment considerations are deployment-critical. However, community fine-tunes (Dolphin, uncensored variants) remove or bypass these guardrails. The Llama system prompt format also affects refusal rates — deviating from Meta's recommended format increases unintended refusals by ~15-20% on [Llama 3.3](/families/llama) Instruct per community testing. Attention-sink behavior manifests in long-context Llama serving. Llama 3+ models exhibit attention sinking (first token absorbs disproportionate attention weight), causing quality degradation on information at extreme positions in very long contexts (>64K). This is less severe than in some other architectures but measurable — critical information should be placed early in context for Llama-based applications. Quantization-specific deployment bugs exist. This was fixed in vLLM 0.6.3 but illustrates that even mature quantization paths have version-dependent correctness. Use AWQ for structured generation deployments on Llama.