Foundations & Models
Pretraining
In pretraining, the model reads once across everything in the way of text and data that can be rounded up, and learns language, facts and patterns along the way — without a task, without a goal, just prediction. This phase takes the lion's share of the millions and determines the character of the model. Everything later — fine-tuning, alignment — is finishing work on a blank that is created here.
An example
In pretraining, the model gets no tasks and no evaluation by humans. It gets text — a great deal of text — and the same exercise over and over: hide the next word and guess it. Billions of times. From this dull repetition emerges, as a side effect, everything that later looks like knowledge: grammar, connections between facts, programming conventions, manners. Nobody taught them one by one; they were the cheapest way to guess the next word better.
A closer look
In pretraining, the model learns from very large, mostly unlabeled amounts of text by continually predicting missing pieces. This is where a feel for language and world knowledge arise, and where the largest share of the costs is incurred.
Why it matters
Two practical consequences. First: what was not in the training texts, the model does not know — your internal pricing model, your works agreement, last week's case. No prompt makes up for that; it has to be supplied. Second: pretraining has a cutoff date. Everything after is missing, and the model does not know it is missing — it answers about recent events as fluently as about old ones. For anything time-critical, a model without a connection to current sources is the wrong source.
Commonly misunderstood
What did not appear in pretraining can only be supplied later to a limited extent. Internal terms and company-specific connections therefore belong in the context, not in the hope of prior knowledge.