Training & Alignment
RLHF
After pretraining, humans rate model answers: better, worse, not like that. From these judgments the model learns what counts as helpful, harmless, and honest. RLHF turned raw text prediction into the approachable assistants we know. It also transfers the preferences of the raters, however: agreement is rewarded, contradiction rarely.
An example
After pretraining, a model can continue text but not necessarily answer helpfully. So people are asked to compare two answers at a time and mark the better one, hundreds of thousands of times. These comparisons yield a reward model, against which the language model is then optimized. The result is the difference between a text generator and something that behaves like a conversation partner.
A closer look
Humans compare pairs of answers, a reward model is learned from this, and it serves as the target for further training. This is how the helpful tone of today's assistants comes about. It is trained in, not a property of the language model.
Why it matters
Whoever made the comparisons shapes what the model considers a good answer, and these people are not your workforce, not your industry, and usually not your cultural background. A side effect is well documented and noticeable in everyday use: what is rated is what seems helpful, and so models trained this way tend toward agreement. They contradict a false premise in the question less often than they should. For specialist applications where contradiction would be the real value, you have to counter this explicitly in the prompt.
Commonly misunderstood
What gets optimized is what raters liked, not what is true. That explains the tendency toward lengthy agreement: it goes over better in comparisons than a terse “I don't know.”