Foundations & Models
Quantization
The weights are stored more coarsely — fewer decimal places, fewer bits — and the model becomes smaller and faster, usually with barely measurable loss of quality. It sounds technical, but it decides whether a model runs on a laptop, a phone or only in the data center. Quantization is one of the reasons local AI has become realistic.
An example
A 70-billion-parameter model needs roughly 140 gigabytes of memory at full precision, and thus several data-center graphics cards. If you store each number with only 4 bits instead of 16, the same model shrinks to about 35 gigabytes — and runs on a single card or a well-equipped workstation. The answers become measurably, but usually only slightly, worse. For many tasks the difference is not noticeable; for some it is decisive.
A closer look
Instead of 16 bits, the weights are stored in 8 or 4 bits. This reduces memory requirements and bandwidth roughly proportionally and makes models runnable on smaller hardware. The loss of precision is usually negligible at 8 bits and becomes noticeable below that.
Why it matters
Quantization is the knob that turns “only works in the cloud” into “works in-house” — and thus often the technical precondition for data not having to leave the company. But it is not a free gain, and how much quality is lost depends heavily on the task. So do not ask for a general assurance, but for a comparison on your own cases: the same hundred inputs through both versions, results side by side. That measurement costs a day and settles the question for good.
Commonly misunderstood
The loss does not show up in the average but at the edges: long contexts, numbers, rare languages. Anyone who checks only the usual test cases will not see it.