Edge Computing
Edge computing means running AI inference on a local device (laptop, phone, embedded system) instead of sending data to a remote cloud server. For local AI operators, this is the default workflow: the model runs on your own hardware, keeping data private and avoiding network latency. The trade-off is limited compute and memory compared to cloud clusters, which constrains model size and speed. Edge computing matters because VRAM and RAM on consumer GPUs (8–24 GB) or Apple Silicon (8–128 GB unified memory) determine which quantized models fit and how fast they run.
Deeper dive
Edge computing shifts computation from centralized data centers to the 'edge' of the network — the device where data is generated or consumed. In AI, this means loading a model onto a local GPU or NPU and running inference without internet dependency. Operators encounter this trade-off daily: a 70B parameter model at Q4 requires ~40 GB VRAM, which exceeds most consumer GPUs, so you either quantize further (Q2–Q3), offload layers to system RAM (slower), or use a smaller model. Edge inference also affects latency: local runs avoid round-trip times (often 50–200 ms to cloud), but token generation speed depends on hardware (e.g., ~40 tok/s on RTX 4090 vs ~5 tok/s on M1 MacBook Air for 7B models). Frameworks like llama.cpp, Ollama, and MLX are optimized for edge hardware, supporting CPU, GPU, and hybrid offloading.
Practical example
An operator with an RTX 3060 (12 GB VRAM) wants to run Llama 3.1 8B at Q4_K_M (5 GB). That fits comfortably, leaving room for a 4K context window (2 GB). But if they try Llama 3.1 70B at Q4 (~40 GB), the card can't hold it — the runtime must offload most layers to system RAM, dropping speed from ~30 tok/s to ~2 tok/s. This is the edge computing constraint: local hardware limits model choice.
Workflow example
When you run ollama run llama3.1:8b, the model loads entirely into VRAM on an RTX 4090 (24 GB). If you run the same command on a laptop with only integrated graphics, Ollama falls back to CPU inference — you'll see tokens/sec drop from ~80 to ~5. In LM Studio, you can set 'GPU Offload' sliders to control how many layers run on the GPU vs CPU, directly managing edge hardware limits.
Reviewed by Fredoline Eruo. See our editorial policy.