Direct Preference Optimization (DPO)

DPO emerged as a simpler alternative to RLHF by mathematically reformulating the reinforcement learning objective into a standard supervised learning problem. Instead of the three-stage RLHF pipeline (collect preferences, train reward model, run RL), DPO directly optimizes the language model on pairs of preferred and rejected responses in a single training step.

The key insight is that the optimal policy under the RLHF objective has a closed-form relationship with the reward function. This means you can skip the reward model entirely and directly increase the probability of preferred responses while decreasing the probability of rejected ones, with a regularization term that prevents the model from deviating too far from its base behavior.

For teams building aligned AI products, DPO matters because it lowers the barrier to alignment tuning. RLHF requires deep reinforcement learning expertise and significant infrastructure. DPO uses standard supervised fine-tuning tools, making it accessible to any team comfortable with fine-tuning. The trade-off is that DPO can be less effective on complex preference landscapes where the reward model in RLHF would have provided more nuanced guidance.