Compute & Cost
Inference Costs
The price of a single answer, driven by model size, context length, and thinking time. Individually fractions of a cent, at scale a budget item: a thousand employees times dozens of requests a day times 250 working days. Anyone introducing AI seriously needs the same cost accounting for it as for any other resource, per task, not per license.
An example
A transaction sends 4,000 tokens of context and gets 600 back. With a mid-range model, that is about 1.5 cents. Sounds like nothing, until you multiply by 8,000 transactions a day: 120 euros daily, 44,000 euros a year, for a single workflow. If the same transaction runs through an agent with twelve steps instead of one call, the same bill gets a double-digit factor in front of it.
A closer look
Billing is by tokens, usually with different prices for input and output. The biggest lever is rarely the model but the context length: a system that sends ten documents with every request pays for ten documents.
Why it matters
Inference costs are the item that accompanies pilots into production and becomes a problem there, because it grows with usage, not with the project. The three most effective levers are known and rarely pulled early: shorten the context instead of sending everything, use a smaller model for the standard case, and cache recurring requests. All three are laborious to retrofit later. That is why the cost calculation per transaction belongs in the architecture phase, not in the first invoice.
Commonly misunderstood
Prices per million tokens have fallen for years, yet bills have risen. More is used and more context is sent along. Volume eats the price decline.