Edge AI
Definition pending
We've cataloged "Edge AI" but haven't written a full definition yet. Definitions are hand-curated rather than auto-generated, so it takes time to cover every term.
Want this one prioritized? Email us and we'll bump it.
Practical example
Edge AI means running models on device, not in the cloud — on a phone, a Raspberry Pi, a security camera, a tractor in a field with no internet. The constraints: limited power (2–15W vs 700W for a GPU server), limited memory (4–16 GB vs 80–640 GB), and no cooling. Models must be small (<3B params) and quantized (<INT8).
Workflow example
Deploy on edge: (1) convert model to ONNX or TFLite, (2) quantize to INT8 or FP16, (3) test on target device — a 1.5B Q4 model runs at ~5–10 tok/s on a Raspberry Pi 5, usable for simple Q&A. For phones, Core ML (Apple) or NNAPI (Android) provide hardware acceleration. Key metric: first-token latency under 1s, throughput >5 tok/s for usable chat.