Reinforcement learning from human feedback (RLHF)

State of the art (2026)

Training on human preferences is still a standard step in making chat models helpful and safe in tone. The methods vary. Direct preference optimisation (DPO) learns from pairs of preferred and rejected answers, with no reinforcement-learning loop and no separately fitted reward model. GRPO is a reinforcement-learning algorithm. DeepSeek-R1 used it with rewards that can be checked automatically, such as whether a maths answer is right. Feedback can come from people, from another AI model or from automatic checks, and any of these can be paired with different algorithms. Labs do not publish their full training recipes, so we cannot say which methods each leading model uses.

Core thesis

RLHF transforms raw language models into helpful, harmless assistants. Anthropic’s Constitutional AI extends this with AI-generated feedback (RLAIF). Every frontier model uses RLHF or variants (DPO, GRPO, RLAIF). The quality of preference and verification training is a key differentiator between providers – it is a large part of why Claude, GPT and Gemini feel different despite similar architectures. The technique is well-understood but expensive and labour-intensive to execute well, and is now bifurcating into preference alignment for tone/safety and verifiable-reward RL for reasoning.