Mistral

The sliding window attention gave [Mistral 7B](/families/mistral) effective context management at linear complexity in the window size while preserving global attention through periodic full-attention layers (every 6th layer). The 32K vocabulary with sentencepiece BPE emphasized European languages, making it more token-efficient for French, German, Italian, and Spanish than [Llama 2](/families/llama). Mixtral 8x7B (December 2023) introduced MoE: 8 experts with top-2 routing, where each expert is a Mistral 7B-style FFN layer with SwiGLU activation. Total parameters: 46.7B, with 12.9B active per token. The attention layers are shared across experts. Mixtral's shared attention + MoE FFN design made it compatible with every inference engine that supported Mistral 7B. This architecture-compatibility decision was commercially strategic: Mixtral 8x7B inference was trivial to add in engines that already had Mistral 7B support. Mixtral 8x22B (April 2024) scaled the same architecture: 8 experts with top-2 routing, 141B total parameters, 39B active. The FFN hidden dimension and attention heads were expanded proportionally. The architecture was otherwise identical to Mixtral 8x7B, demonstrating Mistral AI's philosophy of architectural conservatism paired with aggressive scaling. Mistral Small (December 2024, 22B dense) and Mistral Medium (March 2025, ~70B dense) continued the dense-architectural line with the same SWA + GQA + SwiGLU recipe at larger scales. Mistral Large was closed-weight and is outside open-weight deployment scope. The Mamba architecture processes tokens sequentially with a recurrent mechanism that has O(1) computation per token regardless of context length. Codestral Mamba demonstrated competitive code generation quality with dramatically lower per-token inference cost for long sequences. This was the first production-quality open-weight Mamba model from a major provider. The architectural evolution from Mistral 7B to Mixtral 8x22B to Codestral Mamba traces a clear strategy: start with lean, efficient dense transformers; extend to FFN-only MoE for capacity scaling; experiment with attention-free architectures for specific use cases. Mistral's design philosophy of minimal architectural complexity and maximal inference compatibility is the fastest path to broad runtime support.

Mistral models are among the most locally-friendly open-weight models at every generation, reflecting Mistral AI's design philosophy of lean, efficient architecture targeting broad deployability. Mistral 7B at release was the most locally-efficient 7B-class model. FP16 required ~15 GB VRAM, fitting on a single [RTX 3060 12GB](/hardware/rtx-3060-12gb) at Q4 with 4 GB to spare. The 7B at Q4 (~4.5 GB) runs on [MacBook Pro M4 Max](/hardware/macbook-pro-16-m4-max) using [MLX-LM](/tools/mlx-lm) at 50+ tokens/second. Mixtral 8x7B (46.7B total, 12.9B active) was the first local-viable MoE model. FP16 required ~94 GB total, but only ~26 GB needs to be in fast memory for active experts. On [Mac Studio M3 Ultra](/hardware/mac-studio-m3-ultra) 192 GB, all 8 experts fit in unified memory and achieve ~20 tokens/second at Q4. This was the most accessible MoE model in 2023-2024, predating [DeepSeek V2](/families/deepseek) by 6 months. Mixtral 8x22B (141B total, 39B active) is the practical ceiling for single-GPU Mixtral deployment. FP16 ~282 GB total, ~78 GB active. Q4 (~80 GB total, ~22 GB active) fits on 2× [RTX 4090](/hardware/rtx-4090) or a single [L40S](/hardware/nvidia-l40s) 48 GB with partial offloading. On [Mac Studio M3 Ultra](/hardware/mac-studio-m3-ultra) 192 GB at Q4, ~12 tokens/second using llama.cpp with expert offloading. The MoE efficiency makes Mixtral 8x22B the most hardware-efficient path to 70B-class capability locally. Codestral Mamba 7B is more locally-friendly than Mistral 7B transformer at long context. At 7B parameters with Mamba architecture, 128K context adds essentially zero VRAM overhead. Codestral Mamba 7B serves 128K context on a [RTX 4060 Ti 16GB](/hardware/rtx-4060-ti-16gb) using ~5 GB VRAM. For local codebase agents ([Aider](/tools/aider), [Cline](/tools/cline)), Codestral Mamba 7B's context efficiency is a genuine architectural advantage over any transformer model at similar scale. The trajectory: Mistral models maintain consistent local viability across generations. Mistral 7B and Mixtral 8x7B are the canonical recommendations for sub-24GB-VRAM MoE deployment. For hardware: [RTX 4090](/hardware/rtx-4090) for Mixtral 8x7B with full offloading; [Mac Studio M3 Ultra](/hardware/mac-studio-m3-ultra) for Mixtral 8x22B in-memory; any device with 6 GB VRAM for Mistral 7B at Q4.

Mistral 7B (September 2023) debuted with MMLU 60.1% (5-shot), competitive with [Llama 2 13B](/families/llama) (54.8%) at nearly half the parameters. On GSM8K 8-shot, 46.1%; HumanEval pass@1 30.5%. The standout: 81.3% on HellaSwag 10-shot, exceeding [Llama 2 70B](/families/llama) (78.0%), evidence of unusually high training data quality for a 7B model. Mixtral 8x7B (December 2023) was the breakthrough: MMLU 70.6% matched [Llama 2 70B](/families/llama) (68.9%) with only 12.9B active parameters, a 5.4× parameter-efficiency win. GSM8K 58.4%, HumanEval 40.2%. TruthfulQA MC2 at 59.3% placed between Llama 2 70B (65.1%) and Llama 2 13B (56.0%). Mixtral 8x22B (April 2024) pushed MMLU to 77.8% and GSM8K to 78.6%, matching [Llama 3 70B](/families/llama) (79.5% MMLU) at 1.8× fewer active parameters. HumanEval 55.1% was competitive. The multilingual benchmarks showed Mistral's European language strength: French MMLU 75.2%, German MMLU 74.8%, Italian MMLU 74.1%, each 2-4 points above Llama 3 70B. Mistral Small 22B (December 2024) scored MMLU 81.5% and GSM8K 83.2%, matching [Llama 3.1 70B](/families/llama) MMLU at 3× fewer parameters. HumanEval 68.3% at 22B scale was exceptional, exceeding every non-MoE open-weight model below 70B parameters. IFEval strict-prompt 67.2%. Codestral Mamba 7B (July 2025) scored HumanEval 52.4% and MBPP 60.2%, competitive with transformer 7B code models like [CodeGemma 7B](/families/gemma) at substantially lower per-token cost for long sequences. LiveCodeBench 28.3% trailed transformer alternatives. The Mistral benchmark identity: maximizing capability-per-parameter through lean architecture and high-quality training data. Where Mistral wins vs [Llama](/families/llama): parameter efficiency at 7-22B scale, European language benchmarks. Where Mistral trails: frontier capability ceiling, reasoning benchmarks trailing [DeepSeek](/families/deepseek) and [Qwen 3](/families/qwen), and very long context degradation from SWA periodic attention.

Mistral models benefit from the broadest runtime support of any non-Llama architecture because of their architectural conservatism. The Mistral 7B-style transformer is a minimal delta from [Llama](/families/llama) architecture, and the Mixtral FFN-only MoE adds only routing logic on top of existing attention implementations. [vLLM](/tools/vllm) added Mistral 7B support in v0.2.3 and Mixtral MoE support in v0.2.5. vLLM's prefix caching is highly effective for Mistral because the shared attention layers mean all tokens hit the same attention cache regardless of expert assignment. [SGLang](/tools/sglang) added Mistral support contemporaneously. SGLang's RadixAttention tree cache works identically for Mistral as for Llama. Structured generation is well-tested on Mistral, making SGLang strong for Mistral API deployments. [Ollama](/tools/ollama) provides one-command deployment: `ollama run mistral:7b`, `ollama run mixtral:8x7b`, `ollama run codestral-mamba:7b`. Mixtral 8x7B on [RTX 4090](/hardware/rtx-4090) via Ollama achieves ~35 tokens/second. Codestral Mamba requires the mamba backend (llama.cpp Mamba branch), included in Ollama since 0.1.40. [llama.cpp](/tools/llama-cpp) supports all Mistral architectures: Mistral 7B dense, Mixtral MoE with expert offloading, and Mamba via the Mamba backend. Mixtral 8x22B Q4 on [Apple M3 Ultra](/hardware/apple-m3-ultra) with Metal achieves ~12 tokens/second. Codestral Mamba on CPU-only achieves ~15 tokens/second with 128K context without throughput degradation. [TensorRT-LLM](/tools/tensorrt-llm) supports Mistral 7B and Mixtral MoE. Mixtral 8x22B FP8 on 4× [H100 SXM](/hardware/nvidia-h100-sxm) achieves ~12,000 tokens/second at batch 128. Expert-aware tensor-parallel placement minimizes cross-GPU communication. [MLX-LM](/tools/mlx-lm) provides excellent Apple Silicon support. Mistral 7B at Q4 on [MacBook Pro M4 Max](/hardware/macbook-pro-16-m4-max) achieves ~55 tokens/second. Mixtral 8x7B at Q4 on [Mac Studio M3 Ultra](/hardware/mac-studio-m3-ultra) achieves ~25 tokens/second with polished SWA Metal shaders. [ExLlamaV2](/tools/exllamav2) supports Mistral 7B with EXL2 format. Mixtral MoE is not supported as of mid-2026. For single-GPU Mistral 7B, ExLlamaV2 at 4.0 bpw on [RTX 4090](/hardware/rtx-4090) achieves ~45 tokens/second. [Text Generation Inference](/tools/text-generation-inference) supports all Mistral models. TGI's SWA implementation correctly handles periodic global-attention layers since TGI 1.3.

Mistral models quantize well at standard bit widths, benefiting from SwiGLU architecture's smooth activation distributions. The outlier-channel problem that plagues some architectures is minimal in Mistral. GGUF via [llama.cpp](/tools/llama-cpp) provides the broadest quantization path. Mistral 7B at Q4_K_M (~4.5 GB) loses ~0.3% MMLU vs FP16, effectively lossless. At Q3_K_M (~3.5 GB), degradation is ~1.5% MMLU, usable for casual use. IQ3_XXS (~3 GB) loses ~2.5% MMLU. Mixtral 8x7B at Q4_K_M (~25 GB) loses ~0.5% MMLU. At Q3_K_M (~19 GB), ~2.0% MMLU degradation from MoE routing interaction with quantized expert FFN. The recommended config is Q4_K_M on [RTX 4090](/hardware/rtx-4090), fitting in 24 GB with 1 GB overhead. IQ3_XXS (~18 GB) fits on 16 GB VRAM. Mixtral 8x22B at Q4_K_M (~80 GB) loses ~0.6% MMLU. At Q3_K_M (~60 GB), ~2.2% degradation. IQ3_XXS (~55 GB) achieves Q3_K_M-quality at ~8% smaller filesize, recommended for [Mac Studio M3 Ultra](/hardware/mac-studio-m3-ultra) 192 GB with headroom. GPTQ and AWQ support is mature. Mistral 7B GPTQ 4-bit (~4 GB) is lossless on MMLU. AWQ 4-bit provides the same quality. For Mixtral 8x7B and 8x22B, AWQ 4-bit achieves lossless quality. The quality difference between GPTQ and AWQ for Mistral is negligible. EXL2 supports Mistral 7B at arbitrary 2.0-8.0 bpw. At 3.5 bpw (~3.2 GB), ~1.5% MMLU degradation. At 4.5 bpw (~4 GB), indistinguishable from FP16. Per-layer variable bitrate for Mistral is effective. FP8 via [TensorRT-LLM](/tools/tensorrt-llm) and [vLLM](/tools/vllm) is the standard datacenter format. Mistral 7B FP8 and Mixtral FP8 are effectively lossless. FP8 KV-cache quantization saves ~2×. Codestral Mamba 7B quantization is less tested. The Mamba SSM weights have structured sparsity from HiPPO matrix parameterization making uniform quantization slightly more damaging. Preliminary Q4_K_M shows ~0.8% HumanEval degradation vs FP16, higher than transformer equivalents but still acceptable. Practical rule: Mistral 7B survives Q3_K_M (~1.5% loss); Mixtral 8x7B needs Q4_K_M minimum for production (~0.5% loss); Mixtral 8x22B can run Q3_K_M (~2.2% loss) but Q4_K_M is the recommended floor.

Mistral's ecosystem is substantial and Europe-centric, with growing global adoption. As the EU's flagship AI company, Mistral benefits from European government procurement preferences, EU AI Act compliance expertise, and strong French/German/Italian/Spanish language support. HuggingFace (Paris-based) has near-instant support for every Mistral release. Derivative model volume is moderate-to-high (hundreds of fine-tunes). Major derivatives: OpenHermes Mistral (NousResearch instruction-tuning), Zephyr series (HuggingFace H4 team's alignment-tuned Mistral 7B, establishing DPO training as a community standard), Dolphin Mistral (uncensored), and OpenBioLLM (biomedical). Mixtral 8x7B derivatives include Dolphin Mixtral and Norobara 2 Mixtral. Mistral fine-tuning volume is behind [Llama](/families/llama) but ahead of [Qwen](/families/qwen) for Western-language-only models. License clarity is mixed. Mistral 7B, Mixtral 8x7B, Mixtral 8x22B, and Codestral Mamba 7B are Apache 2.0, making them the most commercially-flexible choice among major providers. Apache 2.0 is more permissive than [Llama](/families/llama) Community License, [DeepSeek](/families/deepseek) License, or [Gemma](/families/gemma) Terms. However, Mistral Large, Medium, and Small are closed-weight commercial models, creating a mixed-license architecture. Enterprises must distinguish open-weight Mistral from closed-weight Mistral. The HuggingFace integration ecosystem is exceptionally strong. Mistral AI was founded by researchers with strong HuggingFace ties. Every Mistral model has first-day model cards, transformers library support, and TGI deployment configs. The H4 team uses Mistral 7B as their primary alignment research platform, making Zephyr the most-cited DPO alignment research in open-weight AI. Training data transparency is minimal, comparable to [Llama](/families/llama) and less transparent than [Aya](/families/aya) or [Phi](/families/phi). The EU AI Act may push Mistral toward greater disclosure for EU-marketed models. The European-language ecosystem is growing: French government AI initiatives deploy Mistral, German industry consortia fine-tune for industrial applications, Italian universities use Mistral for Italian instruction-tuning. Mistral's European-language advantage over [Llama](/families/llama) and [Qwen](/families/qwen) is a sustainable differentiator.

Mistral fine-tuning is well-supported due to architectural conservatism. The standard transformer architecture uses the same fine-tuning stack as [Llama](/families/llama) with minimal adaptation. LoRA and QLoRA support is first-class. HuggingFace PEFT supports Mistral with the `mistral` architecture tag. For Mistral 7B, QLoRA rank 64 on [RTX 4090](/hardware/rtx-4090) 24 GB uses ~18 GB VRAM and trains 10K samples in ~3 hours. For Mixtral 8x7B, QLoRA should target the shared attention layers exclusively for best results; targeting expert FFN layers requires routing-aware LoRA placement. Recommended Mixtral QLoRA: rank 32, target Q/K/V/O attention projections only, 4-bit base model, on [RTX 4090](/hardware/rtx-4090) using ~22 GB VRAM, ~8 hours for 10K samples. Full fine-tuning: Mistral 7B needs ~56 GB VRAM (1× [A100 80GB](/hardware/nvidia-a100-80gb-sxm) or 2× [RTX 4090](/hardware/rtx-4090) with ZeRO-2). Mixtral 8x7B needs ~200 GB VRAM (3× [A100 80GB](/hardware/nvidia-a100-80gb-sxm) or 2× [H100 SXM](/hardware/nvidia-h100-sxm) with ZeRO-3), making deep Mixtral fine-tuning cost-prohibitive for individuals. Sliding window attention impacts fine-tuning: tokens beyond 4096 positions apart have zero direct attention until periodic global-attention layers. For code fine-tuning with long-range dependencies, [DeepSeek Coder V3](/families/deepseek) with full global attention is more appropriate. The dataset ecosystem includes standard instruction datasets plus European-language-specific datasets. The H4 Zephyr alignment recipes are documented for Mistral 7B, providing a fully reproducible DPO alignment pipeline. Axolotl provides pre-built configs for Mistral 7B and Mixtral 8x7B with correct chat template handling (Mistral uses `[INST]` and `[/INST]` tokens). Unsloth supports Mistral 7B (~2× speedup) but not Mixtral MoE. The most common Mistral fine-tuning mistake is using the Llama template on a Mistral model. Codestral Mamba 7B fine-tuning is emerging. The Mamba architecture requires Mamba-specific PEFT adapters; the mamba-ssm library provides experimental LoRA but is not production-mature. Mamba fine-tuning is at the research stage. DPO alignment via the canonical Zephyr recipe (beta=0.1, learning rate 5e-7, 3 epochs on UltraFeedback) produces strong results. Community Mixtral DPO variants achieve 3-5 MMLU point improvements from alignment alone.

Mistral's Apache 2.0 license for open-weight models is the simplest license situation in open-weight frontier AI. No MAU threshold, no commercial-use restriction, no use-case enumeration, just attribution and patent grant. For enterprises needing the simplest license assessment, Mistral open-weight models are the lowest-risk choice. Verify which Mistral model you're deploying. Sliding window attention limits direct attention to a 4096-token window. Retrieval accuracy drops measurably beyond 16K-token separation. For RAG deployments, place retrieved documents close to the query position. This is less severe for Mixtral, whose 56 layers provide more global-attention hops than Mistral 7B's 32 layers. Mixtral MoE's top-2 routing is simpler to serve than [DeepSeek](/families/deepseek)'s top-8 but produces less balanced expert load. Under bursty traffic, certain experts overload while others idle, manifesting as latency spikes. Mitigate by increasing batch size or using [vLLM](/tools/vllm)'s EPLB expert-parallel load balancing. Codestral Mamba 7B deployment has unique considerations. Production Mamba monitoring, batching, and autoscaling tooling expects transformer architectures. The SSM state must be primed through the full sequence before quality generation, producing a different time-to-first-token profile. Chat template accuracy is critical. Mistral uses `<s>[INST]` and `[/INST]` tokens. Using [Llama](/families/llama) or ChatML templates degrades IFEval strict-prompt by 10-15%. The system prompt format concatenates system and first user message into a single instruction block; misformatting produces incoherent instruction-following. Mistral's tokenizer is ~10% more efficient for European languages than [Llama 3+](/families/llama) but ~15% less efficient for CJK languages. For non-European deployments, [Qwen](/families/qwen) provides dramatically better CJK tokenizer efficiency. For production Mixtral serving with [vLLM](/tools/vllm), the recommended deployment topology is 2× [L40S](/hardware/nvidia-l40s) with tensor parallelism + expert parallelism for Mixtral 8x22B, or a single [L40S](/hardware/nvidia-l40s) for Mixtral 8x7B. A known edge case: Mistral 7B Instruct has been observed to produce repetitive completions at temperature 0 with certain system prompts. Setting temperature to 0.1 resolves this while preserving determinism. This behavior is specific to post-RLHF Mistral 7B Instruct and does not affect Mixtral or base models.