Training
Direct Preference OptimizationDPO
Also known as: preference optimisation
A simpler alternative to RLHF that tunes a model directly on "this answer is better than that one" comparisons, without a separate reward model or reinforcement learning.
DPO is a cleaner way to teach an AI good behaviour from human preferences. Like RLHF, it uses examples of "answer A is better than answer B", but it skips RLHF's complicated extra steps and trains the model directly on those comparisons. The result is similar quality with a much simpler, more stable process, which is why it became popular for aligning open models.
Direct Preference Optimization (DPO) is a method for aligning a language model to human preferences using pairs of responses labelled "preferred" and "rejected". It achieves what RLHF does, tune the model toward what people prefer, but without training a separate reward model or running reinforcement learning.
Why it caught on
- Simpler: it is essentially a special fine-tuning step, easier to implement.
- More stable: avoids the instability of RL optimisation.
- Effective: matches or approaches RLHF quality on many tasks.
It is part of the broader alignment toolkit that turns a raw model into a helpful, well-behaved assistant.
DPO reframes preference alignment as a direct classification-style objective on preference pairs, exploiting a closed-form relationship between the optimal RLHF policy and the reward to optimise the policy directly, no explicit reward model, no RL loop (Rafailov et al. 2023). It optimises the log-likelihood ratio of preferred vs rejected completions relative to a reference model, with an implicit KL constraint, recovering much of RLHF's benefit with greater simplicity and stability.
Trade-offs and context: DPO is offline and sensitive to the preference-data distribution and to the reference model; it can be less flexible than online RL for iterative improvement, and can over-optimise or degrade diversity if unregularised. It spawned a family of offline preference methods (IPO, KTO, ORPO) and is frequently combined with, or alternated against, online RLHF in production alignment pipelines. Both approaches shape behaviour and preference-conformance, not factual grounding, so neither removes hallucination.