Dense Model

A dense model activates every parameter on every forward pass — the default architecture for transformers like Llama, Qwen, and Mistral 7B. Distinct from sparse / Mixture-of-Experts models, which route each token through a subset of expert sub-networks.

Dense models are simpler to serve (no routing overhead, simpler quantization paths, no all-to-all communication) but pay full compute cost at inference. A 70B dense model does ~70B parameters × 2 FLOPs × tokens of work per forward pass.

For local AI, dense remains the dominant deployment shape because tooling support is universal and a single dense model fits a single GPU more cleanly than an MoE of equivalent quality.

An operator choosing between Mistral 7B (dense) and Mixtral 8x7B (MoE, ~13B active parameters) for a single-GPU deployment has to weigh VRAM against compute differently for each. The dense 7B model at Q4_K_M needs roughly 4-5 GB and runs every one of its parameters on every token — predictable, simple to reason about, and it quantizes cleanly with any GGUF tool. Mixtral needs VRAM for all 8 experts (quality per active-parameter is better) but only computes through 2 of them per token, so it's cheaper per-token than a dense 47B-parameter model would be despite similar total size. For a memory-constrained single-GPU box where total VRAM is the binding constraint rather than compute, the dense model is often the simpler, more predictable choice — no routing surprises, no per-expert quantization tuning.