Compute & Cost
Edge AI / On-Device
The model runs on the device itself (phone, laptop, machine) instead of in the cloud: no transmission of sensitive data, no dependence on the network, no latency to a remote server. Smaller models and specialized chips make it possible. The division of labor of the future is hybrid: everyday tasks locally, heavy ones in the cloud.
An example
An inspection device in manufacturing detects surface defects right at the line, without a network connection. The reason is not privacy but physics and availability: at eighty parts per minute, there is not enough time for a server round trip, and a network outage must not stop the line. The model is considerably smaller than a cloud model and trained for exactly this one task, which makes it equal to the job.
A closer look
Smaller models run on a phone, laptop, or machine. Advantages are response time without a network, no transmission of sensitive data, and no cost per request. The limits are set by memory, heat, and battery.
Why it matters
Four reasons speak for processing on the device, and they differ in strength: response time, resilience, running costs, and privacy. In practice, one of them is usually the real one, while all four appear in the presentation. It is worth naming the real one, because it determines how much loss of quality is acceptable. The costs don't disappear here, they shift: from the vendor's invoice into hardware, distribution, and updating of potentially thousands of devices.
Commonly misunderstood
On the device does not automatically mean privacy-friendly. Many applications do preprocessing locally and send the rest to the cloud anyway. What matters is what actually leaves the device.