Back to Glossary
Edge AI
Synonyms: on-device AI, local AI, on-device inference, offline AI, TinyML
Do not index
Definition
Edge AI runs AI models directly on a user's device, such as a phone, laptop, browser, or camera, instead of sending data to a cloud server for processing. This allows AI features to respond faster, work without an internet connection, and keep sensitive data on the device. It can also reduce cloud computing costs, although performance depends on the device's hardware and the size of the model.
Use cases
Users can't see why a feature works on one phone and not another.
- The missing button: Chrome's built-in model needs 22 GB of free storage plus a strong GPU or 16 GB of RAM. On older laptops, the feature never appears or silently fails.
- The surprise download: The first tap starts a multi-gigabyte model download. With no progress bar, the feature looks frozen.
- Asking a small model big questions: Apple's on-device model has about 3 billion parameters, and Apple says not to rely on it for world knowledge or math. Apps use it for factual questions anyway.
How it's used in practice
- Check availability before showing the feature: Hide it, or offer a cloud version, when the model isn't on the device.
- Ask before downloading, then show progress: Chrome's docs tell developers to report download progress.
- Label where it runs: Say "Runs on your device" for private tasks, and say when a request goes to the cloud.
- Keep tasks small: Summarize, extract, classify, rewrite. Android's team cut one feature's response time from 13 seconds to under 2 by trimming the prompt.
Test on the slowest device you support. On-device speed depends on the chip. Stream partial output so something appears within 1 second, and show progress if a task runs past 10.
Challenges & limitations
- Device spread: The same feature runs at different speeds, or not at all, depending on the hardware.
- Smaller models: They handle short tasks well and struggle with complex reasoning.
- Updates you don't control: Apple's model ships with the OS, so an OS update can change outputs without an app release.
Free resources
- Edge AI vs. Cloud AI (IBM) — speed, bandwidth, and privacy tradeoffs side by side
- Neural Processing Units Overview (IBM) — the AI chips that make on-device models practical
- Meet the Foundation Models Framework (Apple WWDC25) — Apple's API for the on-device model behind Apple Intelligence
