Deep Reinforcement Learning

Deep Reinforcement Learning (DRL) combines deep neural networks with reinforcement learning, enabling agents to learn optimal actions through trial and error in complex environments. In practice, DRL trains a policy network to map states to actions, using rewards as feedback. For operators, DRL is relevant when fine-tuning language models with RLHF (Reinforcement Learning from Human Feedback), where a reward model scores outputs and a policy model (e.g., Llama) is updated via PPO. DRL requires substantial compute—training a policy can take thousands of GPU-hours—but inference is just a forward pass through the policy network.

DRL extends classic RL (Q-learning, policy gradients) with deep neural networks as function approximators. Key algorithms include Deep Q-Networks (DQN) for discrete actions, Proximal Policy Optimization (PPO) for continuous control, and Actor-Critic methods. In LLM fine-tuning, PPO is the standard: a reward model (trained on human preferences) provides scalar rewards, and the policy (the LLM) is updated to maximize expected reward while staying close to the original model via KL regularization. Operators encounter DRL indirectly via RLHF pipelines: they may run reward model inference or PPO training using frameworks like TRL (Transformer Reinforcement Learning) or Axolotl. The compute cost is significant—PPO requires multiple forward passes per token (policy, reference, reward, value), increasing VRAM and time. On consumer hardware, RLHF fine-tuning is often done with LoRA adapters to reduce memory.

An operator fine-tuning Llama 3.1 8B with RLHF using TRL: they first train a reward model (e.g., a 7B parameter model) on preference data, then run PPO training. With a single RTX 4090 (24 GB VRAM), they can load the policy and reward model using 4-bit quantization (QLoRA) and a batch size of 1. Training throughput might be ~500 tokens/second for the policy forward pass, but PPO requires four forward passes per step, dropping effective throughput to ~125 tok/s. A full RLHF run on 10K prompts could take 2-3 days.

In practice, operators run DRL via scripts using the TRL library. A typical command: accelerate launch train_ppo.py --model_name meta-llama/Llama-3.1-8B --reward_model_path ./reward_model --output_dir ./ppo_llama. The script loads both models, applies LoRA adapters, and iterates over prompts. During training, the operator monitors reward scores and KL divergence; if reward plateaus, they adjust the PPO clipping parameter. After training, they merge the LoRA weights and run inference with ollama run or a vLLM server.