MLC LLM

TVM-based LLM compilation framework. Compiles models for any GPU with a Vulkan / Metal / WebGPU / CUDA backend. A cross-platform on-device LLM runtime — runs Llama, Phi, Gemma, Qwen on phones, browsers, and laptops without per-platform rewrites.

Install via pip from the project's wheel index: python -m pip install --pre -U -f https://mlc.ai/wheels mlc-llm-nightly-cu128 mlc-ai-nightly-cu128 (CUDA 12.8 build). The docs list wheels for CPU, CUDA 12.8 and 13.0, and ROCm 6.1 and 6.2, say Vulkan is supported in all Linux packages, and recommend Python 3.13. MLC-LLM works differently from most engines: models need a platform-specific library compiled with Apache TVM, just-in-time on first run by default. mlc_llm chat HF://mlc-ai/Llama-3.2-3B-Instruct-q4f16_1-MLC downloads pre-converted weights, compiles the library and starts an interactive CLI. For server mode: mlc_llm serve HF://mlc-ai/Llama-3.2-3B-Instruct-q4f16_1-MLC --port 8080. The server exposes an OpenAI-compatible API at /v1/chat/completions. Verify: curl http://localhost:8080/v1/chat/completions -H "Content-Type: application/json" -d '{"model":"default","messages":[{"role":"user","content":"Hello"}]}'. For custom models, compile with mlc_llm compile <model-path> --device <target>. First run downloads the weights (about 4.5 GB for an 8B model) and compiles the model library. MLC-LLM also supports WebGPU (browser) and iOS/Android via native runtime.

Best for: cross-platform local inference where the same model must run on phone, laptop, and server, WebGPU browser-based inference (in-browser LLM demos without a backend server), mobile on-device inference with optimized native runtimes, heterogeneous GPU deployments (Intel Arc, AMD Radeon, Qualcomm Adreno, Apple GPU) where CUDA-only engines can't deploy, research and experimentation with model compilation pipelines. Not suited for: rapid model iteration where compilation cost kills velocity (use Ollama or llama.cpp), maximum-throughput NVIDIA datacenter serving (use vLLM), GGUF-based model ecosystems without MLC-format re-compilation, users who need point-and-click setup (MLC-LLM requires compilation awareness).

Use MLC-LLM when you need inference across the widest range of device targets — Windows, Linux, macOS, iOS, Android, and WebGPU (browser) from a single compilation pipeline. MLC-LLM's TVM-based compilation also targets non-NVIDIA hardware (Intel iGPU, Mali, Adreno mobile GPUs). Switch to llama.cpp when you want model loading without a compilation step; MLC-LLM compiles a model library, just-in-time by default. Use vLLM for NVIDIA datacenter production serving where throughput matters more than deployment breadth. Use MLX-LM on Apple Silicon for simpler setup — MLC-LLM works on Apple Silicon but requires the compilation step that MLX-LM skips. MLC-LLM's value is one pipeline across devices: the same weights run on a phone, laptop, and server, with a model library compiled per target.

Problem: The wrong GPU backend is picked. Fix: Set the backend with --device: mlc_llm chat HF://mlc-ai/Llama-3.2-3B-Instruct-q4f16_1-MLC --device vulkan for Vulkan, --device metal for Apple, --device cuda for NVIDIA. The weights are shared, and the compiled model library is specific to the device. Problem: Compilation is slow. Fix: --opt sets the optimization flags. O2 is the default, and the docs describe O3 as extreme optimization. For development, use pre-converted weights from the MLC-AI org on HuggingFace and let JIT compilation run once, since the result is cached. Problem: WebGPU browser deployment fails. Fix: WebLLM needs a WebGPU-compatible browser. Its docs point to the latest Google Chrome and WebGPU Report to verify support. Firefox enabled WebGPU on Windows in version 141.