DeepSeek

DeepSeek V2 (May 2024) introduced the Multi-head Latent Attention (MLA) mechanism — the architectural innovation that defines the entire DeepSeek family. V2 used 16 experts per layer with top-2 routing, totaling 236B parameters with 21B active per token. The combination of MLA (KV-cache compression) and DeepSeekMoE (compute sparsity) meant V2 achieved competitive quality at dramatically lower inference cost than dense models. This eliminated the quality degradation that load-balancing loss terms introduce in MoE training. V3's MoE expanded to 671B total parameters with 37B active per token using 256 experts per layer (1 shared + 256 routed, top-8 active). MLA was preserved with 512-dimensional latent KV representations. Training data: 14.8T tokens emphasizing code (16%) and math (8%) alongside general web text. V3's architecture remains the baseline for open-weight MoE reasoning models in 2026. DeepSeek-R1 (January 2025) applied DeepSeek's reasoning distillation pipeline to the V3 architecture. R1 uses reinforcement learning with GRPO (Group Relative Policy Optimization) to develop chain-of-thought reasoning behaviors — the model learns to generate intermediate reasoning steps without explicit step-by-step supervision. The R1 distillation produced the widely-deployed distilled variants (R1-Distill-Llama-8B, R1-Distill-Qwen-7B/14B/32B/72B), which apply R1's reasoning patterns to smaller dense base models. MoD reduces per-token FLOPs by ~15% without quality degradation. V4 at trillion-parameter total scale (exact parameter counts undisclosed) with ~37-50B active represents the current frontier of open-weight reasoning. The architecture evolution from V2→V4 is unusual in its conservatism: the core architectural primitives (MLA, DeepSeekMoE, GRPO-based RL) remain stable; each generation adds scale, training data quality, and training methodology improvements rather than architectural revolutions. [DeepSeek Coder V3](/models/deepseek-coder-v3) (June 2025) applies the V3 architecture with code-specialized pretraining (85% code data in pretraining mix) and FIM (fill-in-the-middle) training at 50% FIM rate. DeepSeek Coder V3 at full scale matches V3's 671B/37B profile but with code-tuned weight distribution. The key detail: Coder V3 uses repo-level context understanding (file-level dependencies encoded in training) that standard [DeepSeek V3](/models/deepseek-v3) lacks.

DeepSeek models are the least locally-friendly frontier models per raw parameter count but paradoxically among the most locally-viable per unit of capability due to MoE efficiency. The tension: 671B total parameters vs 37B active per token. DeepSeek V2 236B MoE (21B active) was runnable on consumer hardware at release. Q4 required ~130 GB total but only ~15 GB active in GPU memory using expert offloading. [Mac Studio](/hardware/mac-studio-m3-ultra) 192 GB could keep all 256 experts in unified memory, achieving ~10 tokens/second at Q4. This was the first time a frontier-tier MoE model was accessible to individual developers, predating [Qwen 3 MoE](/families/qwen)'s local viability by a year. DeepSeek V3 671B MoE (37B active) shifted the accessibility curve upward. FP16 requires ~1.34 TB — datacenter-only. Q4 (~380 GB) still requires 5× [H100 SXM](/hardware/nvidia-h100-sxm) or 2× [MI300X](/hardware/amd-mi300x) for in-memory deployment. On [Mac Studio M3 Ultra](/hardware/mac-studio-m3-ultra), V3 at IQ3_XXS (~240 GB total, ~45 GB active) achieves ~3-4 tokens/second — functional but not interactive quality. This is the frontier-local ceiling for individual developers. DeepSeek V4 pushes local viability further. At FP8, V4 requires ~1 TB total — 8× [H100 SXM](/hardware/nvidia-h100-sxm) minimum for in-memory deployment. V4 on [Mac Studio M3 Ultra](/hardware/mac-studio-m3-ultra) is theoretical (~2 tokens/second) — practically not usable. DeepSeek R1-Distill-Llama-8B is the most locally-viable DeepSeek reasoning model. On [MacBook Pro M4 Max](/hardware/macbook-pro-16-m4-max), ~25 tokens/second via [MLX-LM](/tools/mlx-lm). The distilled variants are the pragmatic local path for DeepSeek reasoning capability: you get ~70-80% of V3's reasoning quality on specific benchmarks at ~1/100th the compute requirement. DeepSeek Coder V3 at the full 671B scale has the same local viability profile as V3. However, Coder V3's distilled variants (not yet released as of mid-2026) would follow the same pattern. The code-specialized 16B/34B distillations (hypothetical) would be the sweet spot for local code assistants. The trajectory: DeepSeek's frontier models are getting harder to run locally (V2→V3→V4 each increases total parameters) but the distilled models are getting better (R1-Distill-Llama-8B is more capable than would be expected of an 8B model, due to reasoning distillation).

DeepSeek V2 (May 2024) debuted with MMLU 80.2% (5-shot), GSM8K 80.4% (8-shot), and HumanEval 68.9% (pass@1) — competitive with [Llama 3 70B](/families/llama) while activating only 21B parameters. The math benchmarks were the signal: MATH 500 at 56.3% (5-shot) exceeded every open-weight model at release. MMLU-Pro (harder subset) at 52.4% matched GPT-4-0314. The Chinese benchmarks (C-Eval 83.0%, CMMLU 84.2%) matched [Qwen 2 72B](/families/qwen) — DeepSeek was competitive from launch on both English and Chinese. DeepSeek V3 (December 2024) was the inflection point. The LiveCodeBench score of 52.7% made V3 the best open-weight code model at release, exceeding [DeepSeek Coder V3](/families/deepseek) predecessor. IFEval strict-prompt at 82.3% matched GPT-4-Turbo. On AIME 2024 competition math, V3 scored 39.8% — the first open-weight model to score meaningfully on competition-level math. GPQA Diamond 59.1% matched Claude 3.5 Sonnet. MMLU-Pro 75.9% exceeded GPT-4o (72.6%). DeepSeek-R1 (January 2025) pushed reasoning-specific benchmarks dramatically. AIME 2024 reached 79.8% (pass@1, majority voting 64 samples) — competitive with OpenAI o1-preview on competition math. MATH 500 at 97.3% (pass@1). GPQA Diamond 71.5%. On Codeforces (competitive programming), R1 achieved 96.3-th percentile. R1-Distill-Qwen-32B scored MMLU 86.1% and MATH 500 89.0% — matching [Llama 3.1 70B](/families/llama) math scores at <50% active parameters. DeepSeek V4 (March 2026) continues the upward trajectory. Known benchmarks: MMLU ~91%, GSM8K ~97%, AIME 2024 ~85%, GPQA Diamond ~78%. LiveCodeBench ~65-70%. These numbers position V4 as competitive with GPT-5 and Claude 4 in open-weight form. Against [Qwen 3 235B-A22B](/families/qwen): V4 wins on math and code benchmarks by 5-10 points; Qwen 3 wins on multilingual benchmarks by 5-10 points. Where DeepSeek wins vs peers: math, code, and competition-level reasoning. This is the family's benchmark identity — DeepSeek models are designed and measured primarily on mathematical reasoning and code generation, with general knowledge benchmarks as secondary metrics. Where DeepSeek trails: multilingual coverage beyond Chinese-English (weaker than [Qwen](/families/qwen) and [Aya](/families/aya)), long-form writing quality assessments, and safety/constitutional benchmarks (DeepSeek models score lower on harmfulness refusal tests than Western-aligned models).

DeepSeek models present a bifurcated runtime support landscape. The MoE architecture with MLA attention requires custom inference code in every engine; support is narrower than [Llama](/families/llama) but deeper where it exists. The distilled dense variants (R1-Distill-Llama/DeepSeek-Coder-V3) require zero custom code — they are standard dense architectures using any engine. [vLLM](/tools/vllm) is the primary production serving engine for DeepSeek MoE. vLLM 0.6.0+ added DeepSeek-V2/V3 architecture support with MLA attention kernel, expert parallelism, and MoE-specific prefix caching. vLLM's PAGE_ATTENTION_V2 backend is required for MLA performance — PAGE_ATTENTION_V1 does not support the compressed KV-cache format. For DeepSeek V4, vLLM 0.7.0+ added Mixture-of-Depths routing support. [SGLang](/tools/sglang) added DeepSeek MoE support in v0.2.5 with the DeepSeek MLA kernel contributed by the DeepSeek team. SGLang's deterministic generation (constrained decoding) is correctly implemented for DeepSeek, which matters for code generation use cases. [llama.cpp](/tools/llama-cpp) supports DeepSeek V2/V3 MoE architecture from b2710+ with expert offloading. The MLA attention implementation in llama.cpp is a CPU-side kernel — GPU-accelerated MLA is not available in llama.cpp as of mid-2026. This means llama.cpp DeepSeek MoE inference is CPU-memory-bandwidth-bound. [Ollama](/tools/ollama) uses llama.cpp as backend and benefits from the same MLA support. Full-scale V3/V4 on Ollama is possible through custom GGUF import but not practical for interactive use — the ~4 tokens/second decode speed with expert offloading makes conversation-style interaction frustrating. Ollama is best for DeepSeek distilled variants. [TensorRT-LLM](/tools/tensorrt-llm) added DeepSeek architecture support in TRT-LLM 0.12.0 with custom MLA + MoE plugins. The build time is significant: constructing the TRT engine for V3 takes ~2 hours on 8× H100. Worth the investment for sustained production but impractical for experimentation. [ExLlamaV2](/tools/exllamav2) does not support DeepSeek MoE (MLA + MoE combination is not implemented). Use vLLM or SGLang for NVIDIA GPU DeepSeek serving. The distilled variants (R1-Distill-Llama-8B, R1-Distill-Qwen-7B/14B/32B) run on every engine that supports the base architecture — which is every engine. The runtime support ecosystem for DeepSeek is effectively "full support for distilled variants, narrow but production-quality support for full MoE models."

DeepSeek MoE quantization presents unique challenges because of the interaction between MLA attention compression and weight quantization. The MLA mechanism already compresses KV representations through low-rank projection — adding quantization on top requires careful calibration to avoid compounding approximation errors. GGUF via [llama.cpp](/tools/llama-cpp) provides the broadest quantization support for DeepSeek MoE. At Q3_K_M (~280 GB), degradation reaches ~3.5% MMLU, which crosses into noticeable quality loss. Below Q3, DeepSeek MoE quality degrades rapidly: Q2_K loses 8-12 MMLU points, making the model functionally broken. The distilled variants (R1-Distill-Llama-8B, R1-Distill-Qwen-7B/14B/32B) quantize identically to their base architectures. R1-Distill-Llama-8B at Q4_K_M loses ~0.5% MMLU — standard Llama quantization numbers. R1-Distill-Qwen-32B at Q4_K_M loses ~0.6% MMLU — standard Qwen numbers. The reasoning fine-tuning does not measurably impact quantization sensitivity for these dense distillations. AWQ and GPTQ support for DeepSeek MoE is narrow. GPTQ has partial support through community adapters but is not production-grade. There is no production-quality 4-bit GPU quantization for DeepSeek MoE that matches the quality of [Llama](/families/llama)'s AWQ/GPTQ/EXL2 ecosystem. FP8 via [TensorRT-LLM](/tools/tensorrt-llm) and [vLLM](/tools/vllm) v0.6.3+ is the recommended production format. DeepSeek V3 FP8 (~670 GB) is effectively lossless: MMLU degradation <0.1%, GSM8K within noise. FP8 KV-cache compression (saving ~2× on KV-cache memory) is also supported for DeepSeek MoE in vLLM with MLA-aware KV-cache quantization. The router is a small MLP (~75M parameters in V3) and should never be quantized below FP16. Expert weights can be quantized to Q4 without routing issues, but the router must stay at full precision. This is nearly identical to the Qwen MoE routing constraint despite the different routing algorithms (learned vs hash-based). This fits in 192 GB unified memory by keeping only active experts resident and swapping inactive experts to SSD. Expect ~3 tokens/second at IQ3_XXS quality (~2.5% MMLU degradation vs FP16) — the tradeoff is quality for local accessibility. For most users, the distilled 32B variant at Q4_K_M on [Mac Studio](/hardware/mac-studio-m3-ultra) with ~15 tokens/second and ~1% degradation is the superior local experience to a severely quantized full DeepSeek MoE.

DeepSeek's ecosystem is younger than [Llama](/families/llama) or [Qwen](/families/qwen) — the company is smaller, model releases are more recent, and DeepSeek's custom architecture (MLA + MoE) creates a higher barrier to community fine-tuning than standard dense models. Despite these headwinds, the ecosystem has grown rapidly around DeepSeek's core value proposition: frontier reasoning capability at disruptive pricing. Derivative model volume is moderate compared to Llama (hundreds of derivatives, not thousands). The distilled variants have a richer fine-tuning ecosystem because they are standard 7B-34B dense models with standard fine-tuning requirements. The training-data and methodology transparency is a DeepSeek differentiator. The V3 paper (249 pages) documents the auxiliary-loss-free load balancing algorithm in sufficient detail for reproduction. The R1 paper documents the GRPO reinforcement learning pipeline. This transparency has spawned active replication efforts — several teams have reproduced the R1 reasoning pipeline on non-DeepSeek base models. License clarity is strong with one caution: DeepSeek's license is permissive for research and commercial use, allows redistribution and modification, and does not have a MAU threshold. It is not Apache 2.0 or MIT — the specific license text should be reviewed for your deployment. The license is functionally similar to [Llama](/families/llama)'s Community License in practical terms: broad commercial use, redistribution allowed, some use-case restrictions. The Chinese-language ecosystem around DeepSeek is substantial: the models are hosted on ModelScope, integrated with Chinese cloud platforms, and have documentation in Chinese. The global HuggingFace ecosystem is fully active with DeepSeek models, community discussions, and third-party deployment guides. The Discord/Reddit developer community around DeepSeek is active and growing — particularly around V3 self-hosting and R1 reasoning reproduction experiments. DeepSeek models have weaker function-calling performance (measured by Berkeley Function Calling Leaderboard) than Llama 3.1+. Agent frameworks support DeepSeek primarily through OpenAI-compatible API endpoints rather than native model support — this works for basic use cases but misses model-specific optimizations. The DeepSeek chat template is close to ChatML but not identical, causing subtle formatting issues with some agent frameworks.

DeepSeek fine-tuning splits cleanly into two categories with dramatically different compute requirements: full-scale DeepSeek MoE (671B-1T params) and distilled dense variants (7B-34B). Full-scale DeepSeek MoE fine-tuning is datacenter-scale. The compute cost is in range of $10K-$50K per fine-tune run at market GPU pricing. This effectively restricts full-scale DeepSeek MoE fine-tuning to well-funded teams (labs, enterprises, governments). The DeepSeek team's own fine-tuning is done on their internal cluster architecture, which they've documented in technical reports but not released as public infrastructure. Distilled variant fine-tuning is standard-cost. DeepSeek R1-Distill-Qwen-32B fine-tuning is architecturally identical to [Qwen 2.5 32B](/families/qwen) fine-tuning: QLoRA rank 32 on [RTX 4090](/hardware/rtx-4090) 24 GB with 4-bit base model, ~12 hours for 10K samples. The distillation process has already embedded DeepSeek's reasoning patterns; additional fine-tuning layers domain-specific knowledge on top of the reasoning capability. LoRA and QLoRA tooling is available via standard transformers-based frameworks. Axolotl has pre-built configs for DeepSeek distilled variants with correct chat templates and LoRA target modules. [Unsloth](/tools/unsloth) does not currently support DeepSeek MoE architecture (MLA attention is not implemented in Unsloth's fused kernels) but supports distilled variants via their base architecture paths. RLHF/DPO support is available via TRL for distilled variants and the deepspeed-based pipeline for MoE. The GRPO algorithm that DeepSeek used for R1 training is available through community implementations (GRPOTrainer in TRL and independent PyTorch implementations). The recommended practice: 60% domain-specific data + 40% reasoning-chain data to preserve reasoning capability while adding domain knowledge. This is a DeepSeek-specific best practice that differs from standard [Llama](/families/llama) fine-tuning where domain-only fine-tuning is sufficient. Fine-tuning datasets that model these reasoning patterns must match the R1 reasoning style (e.g., using <think> tags, verification markers) or risk confusing the model's reasoning pre-training. This is an emerging fine-tuning methodology challenge as reasoning-distilled models become more common.

DeepSeek's license is permissive for commercial use but has specific restrictions around model usage. Review the specific license text for your deployment — DeepSeek's license is a custom document, not a standard open-source license, and legal review is recommended for production deployment. MoE serving complexity is the primary deployment challenge. DeepSeek V3/V4 MoE requires expert parallelism across multiple GPUs for production throughput. Monitor expert utilization in production; the open-source monitoring tooling for DeepSeek MoE expert balance is less mature than equivalent tooling for dense model deployments. MLA attention compatibility with inference engines varies. The compressed KV-cache format in DeepSeek MLA is 8× smaller than standard attention but requires the inference engine to handle the decompress-and-attend operation efficiently. [vLLM](/tools/vllm) and [SGLang](/tools/sglang) have optimized MLA kernels. [TensorRT-LLM](/tools/tensorrt-llm) has the most optimized MLA implementation but only for NVIDIA H100/B200 GPUs. Context length handling in DeepSeek MoE has a sharp edge at 128K. However, beyond 128K, MLA's compression decompression path introduces noticeable perplexity degradation on retrieval benchmarks. The practical limit for DeepSeek V3 in production is 128K context; do not extend beyond this without testing. Censorship and refusal behavior is a sensitive deployment consideration. DeepSeek V3 and R1 have strong content filtering around Chinese political topics (similar pattern to [Qwen](/families/qwen) but implemented through different RLHF methodology). The refusals are more granular than Qwen's — DeepSeek's RLHF target appears to weight different topic categories differently, producing more nuanced refusal patterns. Multi-token prediction (MTP) speculative decoding is built into DeepSeek V3/V4 inference and improves single-stream throughput by ~1.8× when enabled. For production serving on [vLLM](/tools/vllm), MTP should be enabled for single-user or small-team deployments and disabled for high-concurrency API serving where batching already achieves high throughput. The distilled variants (R1-Distill-Llama-8B) inherit the deployment characteristics of their base architectures — Llama deployment caveats apply to Llama-base distillations. The reasoning-chain behavior can be controlled via a `max_thinking_tokens` parameter in some inference engines, limiting the length of intermediate reasoning.