01. Why Compression?

Chapter 1 of 18 · 15 min

KEY INSIGHT

Model compression reduces computational requirements while preserving predictive performance, enabling deployment on constrained hardware without sacrificing functionality. Large neural networks achieve top-performing results across domains, but their size creates deployment barriers. A model with 7 billion parameters requires approximately 14GB of memory just to store weights in float32 format. Inference demands additional memory for activations, gradients during training, and intermediate computations. These requirements exclude deployment on edge devices, mobile hardware, and resource-constrained servers. Compression addresses three core constraints: memory footprint, inference latency, and computational cost. Memory footprint determines where models can run and how many can serve concurrently. Latency matters for interactive applications where response time affects user experience. Computational cost drives cloud infrastructure expenses and battery consumption on mobile devices. Several compression techniques exist with different tradeoffs. Pruning removes weights or neurons entirely, creating sparse models. Quantization reduces the bit-width of weights and activations, from float32 to int8 or lower. Knowledge distillation transfers capabilities from large models to smaller ones. Low-rank factorization approximates weight matrices with smaller ones. Each technique offers distinct compression ratios, latency improvements, and accuracy tradeoffs. The compression ratio versus accuracy tradeoff follows a Pareto pattern. Aggressive compression saves more memory but degrades performance. Moderate compression preserves accuracy while still gaining efficiency. The operator's role is identifying the compression level matching deployment constraints while maintaining acceptable performance. A practical reality emerges: not all compression techniques apply universally. Vision transformers compress differently than language models. Recurrent networks respond to different pruning strategies than attention-based architectures. Understanding why compression works requires examining what information models encode and how compression affects that information. This course examines pruning and distillation as primary compression tools. Pruning removes structural components from trained models. Distillation trains compact models to mimic larger ones. Later chapters cover combining these techniques into optimization pipelines that squeeze maximum efficiency from limited budgets.

EXERCISE

Identify three deployment scenarios where model size creates specific bottlenecks—memory, latency, or compute—and consider which compression technique might address each scenario.