SwiGLU

SwiGLU is a gated feed-forward activation: (W1·x ⊙ swish(W2·x)) · W3, replacing the standard MLP's GELU/ReLU in modern transformers. Used in PaLM, Llama, Mistral, Qwen, DeepSeek.

The gating gives a small but consistent perplexity improvement vs GELU MLP at the same parameter budget. The cost is one extra weight matrix (W2), making SwiGLU FFN blocks ~50% larger in parameter count than GELU MLP — which is why Llama-style architectures size the FFN dimension at ~2.67× hidden instead of the classic 4×.

For inference, SwiGLU adds one more matmul per layer; modern attention/FFN fused kernels handle it natively.

When estimating VRAM for a custom fine-tune, an operator sizing a Llama 3.1 8B model needs to account for the SwiGLU FFN's extra weight matrix — the feed-forward blocks alone (with three matrices instead of two) make up a larger share of total parameters than they would in a GPT-2-style GELU MLP at the same hidden size. This matters when someone tries to shrink a model by naively halving the FFN dimension to save memory: because SwiGLU already runs a narrower ~2.67× multiplier instead of 4×, further squeezing it costs more relative capacity than it would in a classic architecture. Practically, if you're writing a custom inference kernel or a from-scratch quantizer rather than using llama.cpp or vLLM, budget for three GEMMs per FFN layer, not two, when profiling matmul time.'