Nemotron

Nemotron-3 (October 2023) launched NVIDIA's entry into the open-weight LLM space with the Nemotron-3 8B model. Training data: NVIDIA's proprietary corpus emphasizing technical documentation, code, and structured reasoning. The 8B model was positioned as a research release, establishing NVIDIA's approach of hardware-informed model design (the model architecture was tuned for optimal throughput on NVIDIA hardware rather than general-purpose deployment). Nemotron-4 (June 2024) scaled the family dramatically: Nemotron-4 15B, Nemotron-4 340B, and Nemotron-4 Nano (3B). The 340B dense model was NVIDIA's flagship, adopting the same GQA + SwiGLU + RoPE architecture at 340B scale. Training infrastructure: NVIDIA's DGX Cloud with H100 clusters, using NeMo Framework for distributed training with model parallelism, tensor parallelism, and pipeline parallelism. The training data emphasized NVIDIA's curated datasets with a focus on high-quality reasoning, math, and code. The alignment pipeline: supervised fine-tuning → reward modeling → PPO optimization → Direct Preference Optimization (DPO) refinement. This multi-step alignment produced instruction-following quality competitive with GPT-4-0314 on the AlpacaEval 2.0 benchmark. Nemotron-3 Nano (February 2025) is the edge-deployment variant: a sub-1B parameter model designed for on-device and embedded deployment. Architecture: dense transformer with GQA at 4 KV heads, SwiGLU, RoPE. The Nano is NVIDIA's contribution to the sub-1B frontier, targeting Jetson edge devices and RTX-powered edge AI. Nemotron-Mini (January 2025, 4B) and Nemotron-Small (March 2025, 14B) filled the size distribution. All Nemotron models share the same core architecture (dense transformer, GQA, SwiGLU, RoPE) at different parameter counts, making them a single architecture family scaled across compute budgets. Nemotron models are designed for peak NVIDIA GPU throughput — the architecture is tuned for TensorRT-LLM's fused kernels, FP8 training/inference, and multi-GPU scaling patterns. This is different from [Llama](/families/llama)'s general-purpose architecture and [DeepSeek](/families/deepseek)'s architecture-first innovation. Nemotron is a hardware-company's model family: the hardware comes first, the architecture is designed for the hardware.

Nemotron models present a bifurcated local viability profile: small models (Nano, Mini, 8B, 14B) are highly locally-friendly; the 340B flagship is datacenter-only. Nemotron-3 Nano (sub-1B) at FP16 (~1.6 GB) runs on any device including smartphones and Raspberry Pi. At Q4 (~0.5 GB), it runs on microcontrollers. This is the edge-AI deployment target for NVIDIA Jetson — Nano is designed for environments where 1 GB of RAM is the total system budget. On a [Google Tensor G4](/hardware/google-tensor-g4) via ONNX Runtime, ~15 tokens/second on-device. Nemotron-3 8B at Q4 (~5 GB) runs on any GPU. On [RTX 4060 Ti 16GB](/hardware/rtx-4060-ti-16gb), ~65 tokens/second via [Ollama](/tools/ollama). On [MacBook Pro M4 Max](/hardware/macbook-pro-16-m4-max), ~45 tokens/second via [MLX-LM](/tools/mlx-lm). The 8B is NVIDIA's recommendation for single-GPU local deployment. Nemotron-Mini 4B at Q4 (~2.5 GB) is the most locally-efficient path to Nemotron-quality. On [RTX 3060 12GB](/hardware/rtx-3060-12gb), ~90 tokens/second. The 4B provides instruction-following quality at embeddable scale. Nemotron-4 14B at Q4 (~8 GB) fits on [RTX 4090](/hardware/rtx-4090) 24 GB with generous headroom. On [Apple M3 Ultra](/hardware/apple-m3-ultra), ~35 tokens/second. The 14B is the practical ceiling for single-GPU local deployment. Nemotron-4 340B is purely datacenter-scale. FP16 requires ~680 GB VRAM — 8× [H100 SXM](/hardware/nvidia-h100-sxm) or 4× [B200](/hardware/nvidia-b200). At Q4 on Mac Studio, ~2 tokens/second — theoretically runnable but practically not usable. Nemotron-4 340B is a cloud/datacenter model for self-hosted deployment on NVIDIA DGX infrastructure. For individual developers, the 8B, 14B, and Nano variants are the local path; the 340B is for enterprise NVIDIA GPU clusters. The trajectory: Nemotron has expanded both downward (Nano at <1B) and upward (340B at frontier scale) while maintaining the 8B-14B sweet spot for local deployment. The Hardware-aware training means Nemotron models achieve 10-15% higher throughput on NVIDIA hardware than architecturally-equivalent non-NVIDIA-optimized models at the same parameter count. For hardware recommendations: [RTX 4060 Ti 16GB](/hardware/rtx-4060-ti-16gb) for Nemotron-Mini 4B and Nemotron-3 8B; [RTX 4090](/hardware/rtx-4090) 24 GB for Nemotron-4 14B; [H100 SXM](/hardware/nvidia-h100-sxm) × 8 for Nemotron-4 340B; NVIDIA Jetson Orin for Nemotron-3 Nano edge deployment.

Nemotron-3 8B (October 2023) scored MMLU 64.8% (5-shot) and HumanEval 32.5% — modest scores reflect its research-release positioning. GSM8K 48.3% was comparable to [Llama 2 7B](/families/llama). The benchmark profile showed an early-stage model family with typical 8B-class capability. Nemotron-4 340B (June 2024) was the breakthrough: MMLU 85.1% (5-shot), GSM8K 88.6% (8-shot), HumanEval 72.3% (pass@1), MATH 500 63.2% — matching [Llama 3 405B](/families/llama) on MMLU within 0.1 points and exceeding it on HumanEval. The alignment benchmarks were the standout: AlpacaEval 2.0 length-controlled win rate 55.2% vs GPT-4 (competitive with GPT-4-0314), Arena-Hard score 48.3%, MT-Bench 8.41. IFEval strict-prompt 80.1% — matching [Llama 3.1 70B](/families/llama) on instruction-following. Nemotron-4 15B scored MMLU 78.2%, GSM8K 82.1%, HumanEval 58.3% — competitive with [Llama 3 70B](/families/llama) at 4.7× fewer parameters. The parameter efficiency was strong but not class-leading (trailing [Phi-3-medium](/families/phi) at similar size). Nemotron-3 Nano scores are modest by design: MMLU 42.3%, HumanEval 18.2% at <1B parameters. The benchmark focus is on-device capability — Nano outperforms similarly-sized models on instruction-following (MT-Bench 4.2 at <1B, competitive with 3B-class models). The Nemotron benchmark identity: competitive general capability at all size tiers with a focus on instruction-following and alignment quality. Where Nemotron wins: instruction-following benchmarks (MT-Bench, AlpacaEval, IFEval) where NVIDIA's HelpSteer2 alignment pipeline produces class-leading results, and throughput-efficiency on NVIDIA hardware. Where Nemotron trails: multilingual benchmarks (behind [Qwen](/families/qwen) and [Aya](/families/aya)), very long context benchmarks (context is 8K-32K, behind [Llama 3.1+](/families/llama) at 128K), and frontier reasoning (behind [DeepSeek V4](/families/deepseek) on math/code). The benchmarks reflect NVIDIA's focus on instruction-following quality over breadth.

Nemotron runtime support is the deepest on NVIDIA hardware and narrower on non-NVIDIA platforms, reflecting NVIDIA's hardware-first design philosophy. [TensorRT-LLM](/tools/tensorrt-llm) is the reference Nemotron inference engine and provides the highest throughput of any engine on any model. Nemotron-4 340B FP8 on 8× [H100 SXM](/hardware/nvidia-h100-sxm) with TensorRT-LLM achieves ~25,000 tokens/second at batch 256 — the highest throughput for any 340B-class model. For production Nemotron deployment, TensorRT-LLM is the recommended engine. [vLLM](/tools/vllm) supports Nemotron since v0.5.0. Nemotron-4 14B on [L40S](/hardware/nvidia-l40s) with vLLM achieves ~6,000 tokens/second at batch 64. vLLM provides competitive but not class-leading Nemotron throughput — the TensorRT-LLM advantage is material for production deployment at scale. NIM is the turnkey deployment path for Nemotron on NVIDIA infrastructure. NIM's Helm charts and Kubernetes operators handle multi-GPU Nemotron serving. For enterprise deployments on NVIDIA GPU clusters, NIM eliminates most of the infrastructure complexity of self-hosting frontier-scale models. [Ollama](/tools/ollama) supports Nemotron through the llama.cpp backend for the 8B and 14B variants. Nemotron-3 8B via Ollama on [RTX 4090](/hardware/rtx-4090) achieves ~50 tokens/second. The 340B on Ollama requires the Mac Ultra path and is not recommended. Ollama is the local deployment path for sub-15B Nemotron models. [llama.cpp](/tools/llama-cpp) supports Nemotron through GGUF. Nemotron-4 14B at Q4 on [Apple M3 Ultra](/hardware/apple-m3-ultra) with Metal achieves ~35 tokens/second. Nemotron-4 340B on llama.cpp is technically functional (expert offloading) but not practically interactive. [MLX-LM](/tools/mlx-lm) provides Apple Silicon support. Nemotron-4 14B at Q4 on [Mac Studio M3 Ultra](/hardware/mac-studio-m3-ultra) achieves ~38 tokens/second. The MLX path is comparable to llama.cpp for sub-15B Nemotron models. [Text Generation Inference](/tools/text-generation-inference) supports Nemotron through the standard HuggingFace pipeline. TGI's integration is functional but not optimized for Nemotron specifically. NVIDIA Jetson platform support is unique to Nemotron Nano: Nano runs on Jetson Orin via TensorRT with ~15 tokens/second at INT8 on 15W power budget. This edge deployment path is exclusive to Nemotron — no other open-weight family has this NVIDIA Jetson integration.

Nemotron quantization is optimized for NVIDIA hardware formats, with TensorRT quantization providing superior results to general-purpose methods. GGUF via [llama.cpp](/tools/llama-cpp) is the standard path for non-NVIDIA deployment. Nemotron-4 14B at Q4_K_M (~8 GB) loses ~0.4% MMLU vs FP16. At Q3_K_M (~6.5 GB), ~2.0% degradation. Nemotron-4 340B at Q4_K_M (~190 GB) loses ~0.6% MMLU; at Q3, ~3.0% degradation. The Nemotron architecture quantizes similarly to [Llama](/families/llama) at equivalent sizes — standard dense transformer quantization behavior. TensorRT-LLM INT8 and FP8 quantization is the recommended path for NVIDIA deployment. Nemotron-4 340B FP8 (~340 GB) is effectively lossless on all benchmarks (<0.1% degradation). INT8 quantization via TensorRT's calibration pipeline achieves <0.3% degradation, better than GGUF Q4 for the same memory footprint. TensorRT's quantization is co-designed with Nemotron's training — NVIDIA uses quantization-aware training techniques during Nemotron's pretraining that improve downstream quantization quality. AWQ and GPTQ support is functional but less optimized than TensorRT. Nemotron-4 14B AWQ 4-bit (~8 GB) achieves lossless MMLU. For NVIDIA GPUs, TensorRT INT8/FP8 is recommended over AWQ/GPTQ for Nemotron — TensorRT's quantization quality and throughput exceed general-purpose quantization methods due to NVIDIA's hardware-aware optimization. FP4 via Blackwell architecture ([B200](/hardware/nvidia-b200)) supports Nemotron with <1% degradation. NVIDIA's Blackwell FP4 microarchitecture support is optimized specifically for Nemotron-scale transformer workloads. NVIDIA provides official quantization tooling: the NVIDIA Model Optimizer (NEMO) for Nemotron quantization, producing calibrated INT8/INT4 engines. NEMO's quantization uses per-tensor + per-channel calibration on representative datasets, achieving ~0.2% MMLU degradation at INT8. This is the recommended quantization path for production NVIDIA deployment. Practical rule: Nemotron-3 Nano and Mini survive Q3_K_M for edge deployment (~2% loss); Nemotron-4 8B and 14B need Q4_K_M for production (~0.5% loss); Nemotron-4 340B should use FP8 via TensorRT for production deployments. On consumer NVIDIA GPUs, AWQ 4-bit is the recommended format. On non-NVIDIA hardware, GGUF Q4_K_M is the standard path.

Nemotron's ecosystem is NVIDIA-centric and enterprise-oriented, with moderate community growth. The ecosystem reflects NVIDIA's position as the dominant AI hardware provider. Derivative model volume is low (dozens of fine-tunes). NVIDIA provides the primary model variants (Nano, Mini, 8B, 14B, 340B) and the HelpSteer2 aligned versions. Community fine-tunes are limited due to the 340B's enormous fine-tuning cost and the smaller models' modest ecosystem momentum. The Nemotron derivatives that exist are primarily enterprise fine-tunes (internal company deployments) rather than public HuggingFace releases. License clarity is strong. NVIDIA Open Model License permits commercial use, redistribution, and modification. The license is permissive and commercially-friendly, comparable to [Llama](/families/llama)'s Community License in practical deployment freedom. There is no MAU threshold. Enterprises already using NVIDIA hardware for AI workloads find the Nemotron license straightforward and well-aligned with their existing NVIDIA agreements. The NVIDIA ecosystem integration is the deepest of any open-weight family. No other model family has this vertical integration from training to edge deployment on a single hardware-software platform. For organizations standardized on NVIDIA infrastructure, Nemotron's ecosystem integration is a compelling advantage over [Llama](/families/llama) (which requires third-party infrastructure) or [DeepSeek](/families/deepseek) (which targets diverse hardware). Training data transparency is moderate. NVIDIA discloses training data categories and the HelpSteer alignment methodology in detail. The actual training datasets are not released. The HelpSteer preference dataset (10M human preference pairs) is partially released for research use. The NeMo Framework ecosystem provides Nemotron-specific training, fine-tuning, and deployment tooling. NeMo's parallelism strategies (tensor, pipeline, expert, sequence) are optimized for Nemotron architectures on NVIDIA DGX infrastructure. For organizations with DGX clusters, NeMo + Nemotron is a turnkey AI stack. Ecosystem gaps: community fine-tuning culture is minimal; non-English fine-tuning community is absent; tools other than NVIDIA's own (vLLM, Ollama, llama.cpp) treat Nemotron as a second-class architecture (standard Llama config with different chat template). Nemotron's ecosystem is deep within NVIDIA's garden and shallow outside it.

Nemotron fine-tuning is best accomplished through NVIDIA's NeMo Framework, which provides Nemotron-optimized training pipelines. Standard open-source tooling also works but lacks NVIDIA's optimizations. LoRA and QLoRA are supported through NeMo and HuggingFace PEFT. NeMo's LoRA implementation for Nemotron uses fused CUDA kernels for 2× training speed vs HuggingFace PEFT on NVIDIA GPUs. For Nemotron-4 14B, NeMo QLoRA rank 32 on [RTX 4090](/hardware/rtx-4090) uses ~16 GB VRAM and trains 10K samples in ~3 hours — faster than HuggingFace PEFT on equivalent hardware. For Nemotron-4 340B, NeMo QLoRA requires ~300 GB VRAM (4× [H100 SXM](/hardware/nvidia-h100-sxm) with NeMo parallelism), making LoRA fine-tuning feasible for enterprise GPU clusters but not for individuals. Full fine-tuning via NeMo: Nemotron-4 14B needs ~112 GB VRAM (2× [A100 80GB](/hardware/nvidia-a100-80gb-sxm) with tensor + data parallelism). Nemotron-4 340B needs ~2.7 TB VRAM (32× [H100 SXM](/hardware/nvidia-h100-sxm) with full parallelism). Full fine-tuning at 340B is a DGX-cluster-scale investment. NeMo Framework provides specific advantages for Nemotron fine-tuning: automatic parallelism configuration (NeMo's auto-parallel analyzes the model + hardware and selects optimal tensor/pipeline/data parallelism), Megatron-Core fused kernels that are specifically optimized for Nemotron's architecture dimensions, and NeMo-Launcher for launching fine-tuning jobs on SLURM/Kubernetes DGX clusters. The dataset ecosystem includes standard instruction-tuning datasets, NVIDIA's HelpSteer preference dataset, and enterprise document corpora. The HelpSteer dataset (10M human preference pairs) is particularly valuable for alignment fine-tuning — it's the dataset NVIDIA used to train the Nemotron reward model, and it's available for community use. Fine-tuning Nemotron with HelpSteer DPO produces measurable alignment improvements. Axolotl has limited Nemotron support; [Unsloth](/tools/unsloth) does not support Nemotron. The recommended fine-tuning path is NeMo for NVIDIA GPU clusters or HuggingFace PEFT for single-GPU experiments. DPO alignment for Nemotron is well-documented through NVIDIA's HelpSteer2 recipe. The multi-step alignment pipeline (SFT → Reward Model → PPO → DPO) is documented in NVIDIA's technical report. Reproducing this pipeline requires significant compute (8× H100 for the full pipeline) but produces class-leading instruction-following quality.

The license is commercially-friendly for standard deployment. This is a standard NVIDIA software license term that may affect organizations building AI platforms on non-NVIDIA hardware. TensorRT-LLM is the strongly recommended inference engine based on Nemotron's hardware-aware architecture design. While vLLM, llama.cpp, and other engines support Nemotron, the throughput difference is 2-3×. This is by design — NVIDIA builds Nemotron for NVIDIA hardware, and the performance gap on alternative platforms is intentional. NVIDIA NIM provides the most turnkey deployment path but adds NVIDIA licensing complexity. NIM containers include pre-built TensorRT engines and are available through NVIDIA NGC. NIM usage requires NVIDIA AI Enterprise licensing for production deployment. Evaluate whether NIM's operational benefits justify the licensing cost vs self-hosted vLLM deployment. Chat template: Nemotron uses a specific format with `<extra_id_0>System {system}<extra_id_1>User {user}<extra_id_1>Assistant {assistant}<extra_id_1>` tokens. This template is non-standard and differs from Llama, ChatML, and Mistral formats. Using other templates degrades instruction-following. The [NVIDIA NIM](/tools/nvidia-nim) deployment path handles template formatting automatically. Nemotron-4 340B deployment at scale requires expert GPU cluster management. The 340B at FP8 requires 4× [H100 SXM](/hardware/nvidia-h100-sxm) with NVLink for tensor parallelism. Deploy on SXM/NVLink variants for the 340B. Edge deployment of Nemotron-3 Nano on Jetson requires NVIDIA JetPack SDK and TensorRT runtime. The Nano model must be converted to TensorRT engine format (trtllm-build) for Jetson deployment. The conversion process is documented in NVIDIA's edge AI documentation. On-device inference on Jetson Orin achieves ~15 tokens/second at 15W power budget — suitable for always-on edge AI applications. A known deployment issue: Nemotron models trained with NeMo Framework use a specific tensor-parallelism-mapped weight layout that differs from HuggingFace's standard layout. Loading NeMo-trained Nemotron weights in HuggingFace requires the NeMo → HF conversion script. This affects self-hosted deployments where the model was fine-tuned with NeMo and then deployed via vLLM/transformers.