COURSE · OPS · A003
RLHF, DPO, and PPO
Learn rlhf, dpo, and ppo through RunLocalAI's practical lens: rlhf, dpo, ppo and alignment, hardware fit, runtime settings, verification habits and local-vs-cloud tradeoffs.
PREREQUISITES
- I003
Why this course matters
RLHF, DPO, and PPO is for operators making local AI reliable, measurable and cheaper to run. It connects rlhf, dpo, ppo, alignment and preference to the questions RunLocalAI wants every reader to answer before they install, upgrade or scale a model: will it run, what will it cost in memory, what setting changes the result, and how do you verify the answer instead of trusting a demo?
What you will be able to do
By the end, you should be able to explain the main tradeoffs in plain language, choose a safe next experiment, and use the chapter exercises as a repeatable operator checklist. The course favors local evidence, hardware fit, context limits, latency and failure modes over generic AI vocabulary.
How to use this course
Start at chapter one if the topic is new. If you already have a working stack, scan for chapters such as Why Alignment?, Preference Optimization Overview, DPO Theory and DPO Implementation with TRL and use those lessons as a quality-control pass before changing a workstation, team workflow or production-like local deployment.
CHAPTERS
- 01Why Alignment?Alignment exists because the training objective (predicting tokens) and the deployment objective (satisfying users) are fundamentally different. Without alignment, models default to imitating the average quality of internet text—which is often not what any individual user needs.15 min
- 02Preference Optimization OverviewPreference optimization separates the "what is good" question (reward modeling) from the "how to be good" question (policy optimization). This separation allows each component to be trained, evaluated, and debugged independently—but also means failures in reward modeling propagate to policy optimization with no intermediate correction.15 min
- 03DPO TheoryDPO makes the reference policy do double duty—it's both the starting point for policy optimization and the implicit baseline for reward. This means the quality of your SFT model directly affects how well DPO can learn preferences. A poor SFT model produces poor reference log-probs, which corrupts the entire alignment process.20 min
- 04DPO Implementation with TRLDPOTrainer handles the reference model automatically, but it does so by creating a separate copy of your model in evaluation mode. This means your effective batch size is halved if you use gradient accumulation to simulate larger batches. Monitor your effective sample throughput during training—if it's much lower than expected, check your gradient accumulation configuration.15 min
- 05DPO HyperparametersThe interaction between beta and learning_rate is critical. A high learning rate with low beta is a recipe for reward hacking—watch for sudden jumps in the reward metric during training. If you see the metric improving too quickly (more than 10% in a single epoch), your learning rate is probably too high.15 min
- 06Reward Model TrainingReward model quality determines the ceiling for your alignment procedure. A poor reward model cannot guide the policy to good outputs regardless of how you optimize. Invest in reward model evaluation before moving to policy optimization—you can catch most problems with targeted tests before wasting compute on RL training.20 min
- 07Data CollectionThe diversity of your preference data matters as much as the quantity. A model trained on preferences for coding tasks will not generalize to creative writing. Map your target use cases and ensure your data distribution matches the deployment distribution.20 min
- 08Reward Model EvaluationA reward model with 60% accuracy on test pairs is barely better than random (50%). But it might still be useful for RL if the errors are systematic and predictable. Conversely, a 70% accurate model with correlated errors might perform worse in RL. Evaluate qualitatively before trusting the numbers.20 min
- 09PPO TheoryPPO's clipping is a conservative mechanism that prevents the policy from "jumping" to a new distribution in a single step. This is crucial when the reward landscape is noisy—as it always is with learned reward models. The KL penalty serves a similar purpose but operates as a soft constraint rather than a hard boundary.20 min
- 10PPO with KL ControlThe target KL is a statement about how much you're willing to let your model change. It's a business/product decision as much as a technical one. A model trained with KL=0.02 will be more faithful to its pretraining but may not align as strongly. Choose based on your use case requirements.20 min
- 11PPO ImplementationPPO training is significantly more complex than DPO and requires more compute. The three-model setup (policy, reference, reward) multiplies your memory requirements. If you can achieve your alignment goals with DPO, prefer it. Reserve PPO for cases where you need fine-grained reward control or when DPO training is unstable.20 min
- 12Synthetic Preference DataSynthetic preference data is most useful for initial alignment and for domain-specific fine-tuning where human data is scarce. For final alignment, human preferences remain the gold standard. Use synthetic data to expand coverage and reduce annotation costs, but validate with human evaluation before production deployment.20 min
- 13Data Quality FilteringData quality filtering is not a one-time preprocessing step—it must be an ongoing pipeline concern. The filtering thresholds that worked for initial training may be inappropriate for later iterations, and adversarial data requires constant monitoring and adaptation.20 min
- 14Iterated TrainingIterated training creates feedback loops between model behavior and data collection. The model shapes what data gets collected, which shapes the next model. Managing this cycle—preventing the model from collapsing into narrow patterns—requires explicit monitoring and intervention at each iteration.20 min
- 15Alignment EvaluationAlignment evaluation is never complete. The model will encounter situations the evaluation did not anticipate. Building reliable alignment requires designing evaluation suites that probe not just known failure modes but the space of potential failures—which means continuous evaluation and iteration.20 min
- 16Helpfulness vs HarmlessnessThe helpfulness-harmlessness tradeoff cannot be resolved with a single parameter—different contexts require different balances. Effective alignment trains models to make context-dependent judgments about when to err on the side of helpfulness versus caution.20 min
- 17Constitutional AIConstitutional AI reduces the human labeling burden by using the model itself to generate feedback. This creates a self-supervised loop where the model learns to critique and improve its own outputs according to human-specified principles.20 min
- 18RRHF and IPORRHF and IPO avoid PPO's complexity by treating alignment as a simpler ranking or classification problem. IPO's explicit regularization makes it more stable, while RRHF's score-based approach is more flexible for multi-response scenarios.20 min
- 19ORPOORPO simplifies alignment training by removing the reference model, reducing memory requirements and training time. The implicit regularization through the style loss term is sufficient to prevent mode collapse in most cases.20 min
- 20Multi-Turn AlignmentMulti-turn alignment cannot be achieved through single-turn training alone. The model must learn to maintain consistent behavior across extended conversations, which requires both training data that includes multi-turn examples and evaluation protocols that test for cumulative failure modes.20 min
- 21Catastrophic ForgettingCatastrophic forgetting is not a bug to eliminate but a fundamental trade-off to manage. Every alignment update will slightly degrade some capabilities. The question is whether the degradation is acceptable for the alignment improvements achieved.20 min
- 22Alignment on Consumer GPUConsumer GPU alignment is feasible with quantization and LoRA, but requires careful batching and memory management. Training times are long but achievable over days rather than weeks.20 min
- 23Benchmarking AlignmentNo single benchmark captures alignment fully. Effective evaluation requires combining multiple benchmarks covering different aspects—safety, helpfulness, honesty, and fairness—and designing custom tests for domain-specific concerns.20 min
- 24Model Alignment Pipeline ProjectThis pipeline demonstrates the complete alignment workflow, but production systems require additional components: continuous monitoring, A/B testing, red-teaming, and iterative improvement based on real-world deployment feedback.25 min