On-Device AI
Definition pending
We've cataloged "On-Device AI" but haven't written a full definition yet. Definitions are hand-curated rather than auto-generated, so it takes time to cover every term.
Want this one prioritized? Email us and we'll bump it.
Practical example
On-device AI runs models locally on phones, laptops, or embedded devices — no cloud API calls, no internet needed. Benefits: zero latency to first token (no network round-trip), complete privacy, works offline. Limitations: small models (<7B params, usually <3B), slower generation (5–30 tok/s vs 50–100+ in cloud).
Workflow example
On-device LLM stack: (1) use Ollama or llama.cpp with a 1.5B–3B Q4 model for laptops (7B if M-series Mac with 16GB+), (2) use MLC Chat or llama.cpp Android for phones, (3) for embedding/search, use ONNX Runtime with 384-dim models (very fast on mobile). Key method: ollama serve exposes local API on port 11434, your app calls localhost. Zero data leaves the device.