Architecture
Transformer
The standard architecture since 2017: the transformer does not process a sequence word by word but looks at all positions simultaneously and learns which are important to one another. That makes it highly parallelizable — and thus trainable at scales where older architectures failed. GPT stands for Generative Pretrained Transformer.
An example
Before 2017, language models read text like a human: word by word, left to right, and what lay twenty words back was almost forgotten. The transformer architecture dropped this sequential order. It processes all the words of a passage at once and lets each one check which others matter for its meaning. That made training parallelizable across thousands of graphics cards — and only thereby made models of today's size affordable at all.
A closer look
The transformer processes all positions of a sequence simultaneously instead of one after another. That made training parallelizable across many graphics cards and was the real breakthrough of 2017: not a new idea about language, but one that was easy to compute.
Why it matters
The architecture explains two properties that stand out daily in operation. First: context is everything. What is in the window, the model can connect; what is not in it does not exist for the answer. Second: the computational effort grows disproportionately with the length of the context, because every word is compared with every other. That is why long contexts are expensive and slow, and why “dump everything in” is not a good strategy even when the window allows it.
Commonly misunderstood
The effort grows quadratically with sequence length. Very long contexts are therefore expensive, and providers buy them with approximations that can lower quality in the middle of the text.