02. Pruning: Unstructured

Chapter 2 of 18 · 10 min

KEY INSIGHT

Unstructured pruning removes individual weights, achieving high sparsity but requiring specialized sparse matrix formats and hardware for speedups. Traditional pruning sets individual weights to zero. The model topology remains intact—layers, connections, activation functions—but most weights become zero. This approach predates modern deep learning, originating from optimal brain damage research in the 1980s. The intuition: many weights contribute minimally to predictions, and their elimination should not degrade performance substantially. Unstructured pruning achieves the highest theoretical sparsity. Compression ratios of 90% or higher are achievable on certain architectures, reducing model weights to 10% of original size. A 1GB model becomes 100MB, dramatically improving memory-constrained deployment scenarios. The implementation challenge lies in storage and computation. Standard dense matrix formats do not exploit zero values—they still allocate memory and perform multiplication. Sparse matrix formats like compressed sparse row (CSR) or compressed sparse column (CSC) store only non-zero values and their indices. However, sparse matrix operations on general-purpose hardware often run slower than equivalent dense operations due to irregular memory access patterns. Three factors determine whether sparse matrices provide speed advantages. First, sparsity level must exceed a threshold—typically 80-90%—where the overhead of storing indices becomes worthwhile. Second, the sparse format must match the hardware's memory access patterns. Third, the hardware must support efficient sparse operations, which current GPUs do not universally. Libraries like `scipy.sparse` provide sparse matrix primitives for Python. Deep learning frameworks offer sparse tensor support with varying maturity. TorchSparse and DeepSpeed provide sparse operations for PyTorch with hardware acceleration on compatible accelerators. A failure mode appears when applying unstructured pruning to batched inference. Batching processes multiple inputs simultaneously, improving hardware utilization. However, each input may have different sparsity patterns, complicating batch-level optimization. This batching inefficiency motivated structured pruning approaches.

Local verification checkpoint

Run the smallest example from this chapter in a local workspace and record the package version, runtime, data path, and observed output. If the result depends on model size, vector count, CPU/GPU backend, or available memory, note that constraint beside the exercise so the lesson remains reproducible.

EXERCISE

Load a small PyTorch model, apply unstructured pruning at 70% sparsity using torch.nn.utils.prune.random_unstructured, and measure the compressed size versus inference speed compared to the dense baseline.