Architecture
Mixture of Experts (MoE)
The model consists of many specialized subnetworks; a router activates only the fitting two or three per request. That way a model can be enormous without every answer having to compute through the whole thing — capability grows faster than cost. Many current top-tier models work this way, even if it is not visible from the outside.
An example
An MoE model with 400 billion parameters perhaps computes with only 40 billion of them per request. A small control component decides for each word which two of a hundred subnetworks are responsible — the rest stay off. The model is thus as large as a very large one and as fast as a medium one. You do, however, have to pay for the memory for all 400 billion, because you do not know in advance which part will be needed next.
A closer look
The model contains many subnetworks, and a selection layer sends each token to two or three of them. As a result the model has a very large number of parameters but computes with only a fraction per request. A model with 400 billion parameters can thus cost as much to run as one with 40.
Why it matters
MoE is the main reason the parameter count has become useless as a cost indicator. With a classic model, you could infer compute effort and price from size; with MoE, size, compute effort and memory requirements diverge. Anyone comparing offers should therefore not ask about parameters but about the three numbers that actually count: price per million tokens, response time under load and memory required when self-hosting.
Commonly misunderstood
Memory requirements do not drop along with it. All experts have to be loaded, even if only a few compute. For self-hosting, that is exactly the expensive part.