Q-Learning

Q-Learning is a model-free reinforcement learning algorithm that learns an optimal action-selection policy by iteratively updating a Q-table, which maps state-action pairs to expected cumulative rewards. The algorithm uses the Bellman equation to update Q-values based on the reward received and the maximum Q-value of the next state. In local AI, Q-Learning is rarely used for LLM inference but appears in fine-tuning pipelines like RLHF (Reinforcement Learning from Human Feedback), where a reward model guides policy updates. Operators may encounter Q-Learning when training custom reward models or agents for tasks like game playing or robotics, often implemented via libraries like Stable-Baselines3.

Q-Learning is a cornerstone of reinforcement learning. The agent maintains a table Q(s, a) representing the expected future reward for taking action a in state s. At each step, the agent selects an action (often using an epsilon-greedy strategy to balance exploration and exploitation), observes the reward r and next state s', then updates: Q(s, a) ← Q(s, a) + α [r + γ max_a' Q(s', a') - Q(s, a)], where α is the learning rate and γ the discount factor. This off-policy algorithm converges to the optimal Q-function under certain conditions. In local AI contexts, Q-Learning is not used for LLM inference but appears in RLHF pipelines: a reward model (trained via supervised learning) provides scalar rewards, and a policy (the LLM) is fine-tuned using PPO, a policy-gradient method that extends Q-Learning ideas. Operators running RLHF on consumer GPUs (e.g., using TRL library) may see Q-Learning referenced in the background.

An operator fine-tunes a 7B LLM using RLHF with PPO. The reward model outputs a scalar reward for each generated response. While PPO is not Q-Learning, it uses a value function (critic) that estimates expected returns, analogous to Q-values. Training on an RTX 4090 (24 GB VRAM) with LoRA adapters, the operator sees ~5-10 tokens/sec generation during PPO rollout collection, and the critic network updates resemble Q-Learning updates.

In a typical RLHF workflow using Hugging Face TRL, the operator runs trl train with a PPO configuration. The script loads a base model (e.g., Llama 2 7B), a reward model, and initializes a value head. During training, the PPO algorithm computes advantages using the value function, which is learned via temporal-difference updates similar to Q-Learning. The operator monitors reward curves and KL divergence in TensorBoard, but does not directly interact with Q-tables.