Compute & Cost
GPU
Originally built for gaming graphics, by chance perfect for AI: graphics processors run thousands of simple operations in parallel, exactly the pattern of neural networks. Nvidia's lead in chips and software (CUDA) made GPUs the scarcest commodity in the industry. Waiting times and prices for these chips are a better economic indicator of AI than any survey.
An example
An ordinary processor computes one thing after another very quickly; a GPU computes thousands of simple calculations at once. Neural networks consist almost entirely of such calculations: multiplying matrices, over and over. That is why the same task runs orders of magnitude faster on a GPU. The accident of history is that this design was originally developed for computer games and AI research repurposed it.
A closer look
Graphics processors compute thousands of matrix operations at once, exactly the operation neural networks are made of. In operation, the limiting factor is rarely compute but memory and its bandwidth.
Why it matters
For decisions about self-hosting, one number matters more than all others: the memory on the card. It determines which model runs at all; if it doesn't fit, no amount of computing power helps. Next comes the question of utilization: a card that is ten percent utilized costs the same as a fully utilized one. That is exactly why self-hosting under fluctuating load is often more expensive than buying, even though the per-request calculation looks better. The comparison must go by utilization, not by list price.
Commonly misunderstood
The number of cards says little. Without suitable interconnects and utilization, expensive hardware sits idle. The metric is utilization, not capacity.