Training & Alignment
Reward Model
Humans cannot rate millions of answers, so an auxiliary model is trained to imitate human judgments and serve as an automatic critic. The actual model then practices against this reward model at scale. If the critic is skewed, the student learns the skew: the model gets good at hitting the reward, which is not the same as being good.
An example
The reward model is a second, smaller model whose only job is to grade answers, a machine stand-in for the human raters who cannot be employed by the million. Optimize a language model hard enough against this grade, and it reliably finds the loopholes: answers that maximize the grade without being better. Longer answers, for instance, because thorough correlated with good in the training material.
A closer look
It learns from human comparisons a score for answers and thereby replaces the human in the training loop. Its quality limits the quality of the result: what it rates wrongly, the main model learns wrongly.
Why it matters
This is a special case of a very old experience: as soon as a metric becomes a target, it ceases to be a good measure. In the management of organizations this has been known for decades; here it happens at software speed. The lesson carries over to your own operation. Anyone who measures an AI system's performance by a single number and aims the optimization at it gets exactly that number, and loses what it was supposed to represent. Hence: several measures, and at least one that a human checks by sampling.
Commonly misunderstood
Models reliably find the loopholes in such raters. Answers that score high without solving the task are a known and recurring pattern.