Compute & Cost
Latency
The wait until the answer arrives: for chatbots a question of comfort, for phone assistants, autocomplete, or real-time systems a question of survival. Latency competes directly with quality: larger models and longer thinking deliver better answers, but later. Good systems therefore choose per task, not across the board.
An example
In a chat, three seconds of waiting goes unnoticed, because the text appears continuously and you read along. In a phone system that answers by voice, three seconds of silence is a broken conversation: people expect under a second there. Same model, same quality, and in one case the solution works, in the other it doesn't. The requirement comes from the use case, not from the technology.
A closer look
Two values count separately: the time to the first token and the rate afterward. For conversations the first is decisive, for long outputs the second. Streaming improves the experience without changing the total duration.
Why it matters
Latency is the requirement most often set too late, and the one that constrains model choice most severely. Reasoning models are good and slow; large contexts are powerful and slow; self-hosting is privacy-friendly and, under fluctuating load, slow. Whoever sets the target time only after building often has to scale back quality. A simple principle helps: the target time is fixed before choosing the model, and derived from the workflow, not stated as “as fast as possible.”
Commonly misunderstood
Agents with many steps add up waiting times. What looks like two seconds individually becomes half a minute over fifteen calls, and goes unnoticed at acceptance because testing happens one call at a time.