Training & Alignment
Overfitting
The model memorizes the training examples instead of the pattern behind them: it shines in the test and fails on anything new. Recognizable by performance dropping sharply on held-out data. The principle is worth keeping as a mental model far beyond AI: processes and metrics, too, can be overfitted to the past.
An example
A model for detecting skin cancer reaches 96 percent in testing. In use it falls to 71. The cause: in the training images, the malignant findings had disproportionately often been photographed with a ruler, because doctors document the size when they suspect something. The model had learned to recognize rulers. In everyday practice without a ruler, it lacks the feature it had relied on.
A closer look
The model adapts so closely to the training examples that it learns their accidents as well. This shows as training error and test error drifting apart: on the familiar it gets better, on the new it gets worse.
Why it matters
Overfitting is why a number from testing is worthless unless it is clear what it was measured on. The simple safeguard is discipline: part of the data is set aside and used only once, at the end. Anyone who looks at this test set repeatedly during development and readjusts accordingly has used it up; the fitting then simply happens by way of the developer. In projects this is the most common reason pilots shine and production disappoints.
Commonly misunderstood
The same thing happens at project level. Anyone who polishes prompts on the same ten examples until all of them come out right has overfitted to ten cases and has not solved the task.