Training
Reinforcement Learning from Human FeedbackRLHF
Also known as: preference tuning
A training method that uses human preferences about which answers are better to make a model more helpful, honest and safe.
RLHF is how a raw AI model is taught good manners and helpfulness. People are shown pairs of the model's answers and pick which is better; the model is then trained to produce more of the kind people prefer. This is the step that turns a clever-but-unruly text predictor into the polite, useful assistant you actually chat with.
Reinforcement Learning from Human Feedback (RLHF) aligns a language model with human preferences. After basic instruction fine-tuning, the process typically goes:
- Collect human comparisons: people rank or choose between model responses.
- Train a reward model that predicts which responses humans prefer.
- Fine-tune the language model with reinforcement learning to score well on that reward model.
Why it matters
RLHF is a big reason modern assistants are helpful, follow instructions, and refuse harmful requests. It shapes behaviour and tone, on top of the knowledge learned in pretraining. It is a core technique in alignment.
Limits
It reflects the preferences (and biases) of the people and guidelines involved, can be gamed by the model, and is costly. Newer methods like DPO achieve similar results more simply.
RLHF fine-tunes a policy (the LLM) to maximise a learned reward model trained on human preference comparisons, classically via PPO with a KL penalty to a reference model to prevent reward over-optimisation and preserve fluency (Ouyang et al. 2022). It is the canonical method for eliciting instruction-following, helpfulness and harmlessness beyond supervised fine-tuning, and a central tool of practical alignment.
Known issues include reward hacking/over-optimisation, sensitivity to annotator guidelines and demographics, sycophancy, and pipeline complexity/instability. This motivated direct preference-optimisation methods, notably DPO, which reparameterise the objective to train directly on preference pairs without an explicit reward model or RL loop (Rafailov et al. 2023), plus RLAIF/Constitutional approaches that substitute AI-generated feedback against written principles to improve scalability and consistency. RLHF and its successors shape behaviour, not factual grounding, so they reduce but do not remove hallucination.