01. Why Alignment?

Large language models are trained to predict the next token. This objective has no concept of helpfulness, harmlessness, or honesty—it simply reflects what text looks like in the training data. When that data comes from the internet, the model learns to imitate everything: sarcasm, misinformation, toxic language, and harmful instructions. Alignment is the process of making model behavior match human values and intentions.

The core problem is distributional shift. A model trained on next-token prediction will eventually be asked to complete prompts it has never seen during training. The model's behavior in these out-of-distribution regions is determined entirely by the patterns it learned—and those patterns may produce outputs that are factually wrong, morally questionable, or simply not what the user wanted.

Three families of alignment techniques address this problem:

Supervised Fine-Tuning (SFT) directly trains on human-written demonstrations of desired behavior. A human labels "good" responses, and the model learns to copy them. This works well but requires expensive human annotation and doesn't scale gracefully—humans must write every type of good response the model might need to produce.

Reward Modeling trains a separate neural network to predict human preferences. Given two model outputs, the reward model predicts which one humans prefer. This learned reward function can then guide the base model toward better outputs without requiring humans to write every response.

Reinforcement Learning from Human Feedback (RLHF) uses the reward model to provide training signal. The most common implementation is Proximal Policy Optimization (PPO), which updates the language model to maximize expected reward while staying close to a reference policy. Direct Preference Optimization (DPO) is an alternative that frames the problem differently, avoiding the need for a separate reward model during the final alignment phase.

The practical reason to care about alignment: a model that says what humans want to hear, explains things clearly, and refuses harmful requests is more useful than an equally capable model that doesn't. Alignment is not about making models "safe" in some abstract sense—it's about making models that actually solve the problems users bring to them.

# The fundamental misalignment: next-token prediction ≠ user satisfaction
# Model sees: "Write a tutorial on hacking"
# Next token prediction: "First, find a vulnerable system..."
# User actually wanted: "Here's how to set up a home lab for learning cybersecurity legally"