Foundations & Models
Parameter
Parameters are the knobs that training adjusts — modern models have billions of them. The number has become a headline because for a long time more parameters meant more capability fairly reliably. Today that holds only with footnotes: data quality, training method and compute at answer time also play a part, and small models beat large ones on narrowly defined tasks.
An example
“70 billion parameters” sounds like a performance figure, like horsepower for an engine. A more useful reading is as space requirements: each parameter is a number that has to be stored and computed through on every request. At common precision, 70 billion of them take up roughly 140 gigabytes — more than fits on a single graphics card. Anyone who wants to know whether a model can run in-house calculates that first, not the quality.
A closer look
Parameters are the numbers that are changed during training. Their count determines how much a model can store at all and, together with the amount of data, how well it does so. For operation, it is also the lower bound of the memory requirement.
Why it matters
The parameter count is a poor indicator of quality and a good one of cost. Smaller, carefully trained models regularly beat larger, older ones; the link between size and ability has long since stopped being automatic. What the number reliably tells you: how much memory is needed, how expensive a request gets and how quickly the answer arrives. For the question “cloud or own hardware?” it is therefore the first metric; for the question “good enough?” it is the wrong one.
Commonly misunderstood
As a basis for comparison, the parameter count has become barely useful. Training data, data quality and post-training explain the differences between current models better than size does.