Ollama

What Ollama actually is

Ollama is a single-binary local LLM server that wraps llama.cpp with a friendly CLI, an HTTP API, and a model registry that auto-resolves slugs like llama3.1:8b to the right GGUF blob. It is not a new inference engine — every token Ollama emits comes out of llama.cpp under the hood. What Ollama provides on top is the operator experience: install, ollama run, get tokens.

If llama.cpp is the engine, Ollama is the dashboard, ignition, and gas pump bundled together. That trade — losing some llama.cpp configurability for radically simpler ops — is exactly why it has become the default first-touch local-AI tool for solo developers and homelabs in 2026.

Where it fits in the stack

Ollama sits at the runtime layer for solo and small-team workloads. The typical stack:

• Frontend: Open WebUI, LibreChat, or any OpenAI-compatible client
• Server: Ollama exposes http://localhost:11434 with a near-OAI-shaped API
• Engine: llama.cpp (bundled — you don't install it separately)
• Hardware: anything llama.cpp supports — CUDA / Metal / ROCm / CPU / Vulkan

It is not the right layer for a multi-tenant production server (use vLLM or SGLang), and it is not the layer to reach for when you need maximum single-stream throughput on a 24 GB consumer card (use ExLlamaV2). It is the layer for "I want a local model behind an OpenAI-compatible URL on this laptop and I want it working in 5 minutes."

Best use cases

• Solo developer / coding agent backend — pair with Continue.dev, Aider, or OpenHands and route inference through http://localhost:11434/v1. See /stacks/local-coding-agent.
• Local chat UI host — pair with Open WebUI; both run as Docker containers on the same homelab host.
• Edge / private inference — running Llama 3.1 8B or Qwen 2.5 7B on an air-gapped workstation.
• Apple Silicon dev box — Metal backend "just works"; no Xcode dance, no MLX-LM Python env to manage.

OS support

If you're on Windows and your ROCm path errors out, see the WSL2 GPU not detected troubleshooting entry.

Hardware / backend support

Ollama inherits llama.cpp's hardware support, which is the broadest in the local-AI ecosystem:

• NVIDIA CUDA — full support, ships pre-compiled CUDA binaries for the most common architectures
• Apple Metal — full support on M-series Macs
• AMD ROCm — supported on Linux with ROCm installed; Windows ROCm support is improving but spotty
• CPU-only — usable for 7B-class quants on a modern desktop CPU; expect 5-12 tok/s decode on a Ryzen 9 7950X
• Vulkan — supported but slower than vendor-native paths; only useful when CUDA/ROCm/Metal are unavailable

Model / quant format support

Ollama is GGUF-only. That's both its biggest strength (one format covers every model in the registry) and its hard ceiling (no AWQ, GPTQ, EXL2, FP8, MLX-4bit). For the deeper picture see /systems/quantization-formats.

The default model registry quants are mostly Q4_K_M — the right balance for most users on most hardware. Use the explicit :q5_K_M / :q8_0 tags when you have spare VRAM or you're running a coding model where the last 2 % of quality matters.

Setup path

The 60-second install:

curl -fsSL https://ollama.com/install.sh | sh
ollama run llama3.1:8b

That's the whole thing on Linux/macOS. On Windows there's a native installer with the same end-state. Inside Docker:

docker run -d -p 11434:11434 --gpus=all -v ollama:/root/.ollama ollama/ollama

If port 11434 is already in use, see Ollama port conflict.

What breaks first

In order of how often you'll hit them:

1. VRAM exhaustion on context-heavy prompts. Ollama doesn't preflight context length the way a hosted API does. Sending a 32K-token prompt to a model that fits 8K of KV-cache on your card will OOM mid-generation. Run with OLLAMA_NUM_GPU and OLLAMA_KEEP_ALIVE set explicitly when this matters.
2. Concurrency. Ollama serves one request at a time per loaded model by default. OLLAMA_NUM_PARALLEL raises the cap, but you're still slower than vLLM at 5+ concurrent users.
3. Custom model imports. Importing a non-registry GGUF works via Modelfile but the syntax is finicky; mismatched chat templates produce silently degraded outputs.
4. AMD ROCm on Windows. Improving but still the most common cause of "Ollama is using CPU and I don't know why" — see ROCm device error.
5. Apple Metal context limits. Models advertise 128K context; Metal will swap unified memory to disk before refusing, and your tokens-per-sec collapses without an obvious error.

Alternatives by intent

Best pairings

• Open WebUI — the canonical chat frontend; both projects target the same operator persona
• Continue.dev — VS Code coding-assistant routed at http://localhost:11434
• Aider — terminal coding agent; Ollama serves the model, Aider drives the edits
• Mem0 + filesystem MCP — memory + tool layer for agentic workflows; see /stacks/local-coding-agent

Who should avoid Ollama

• Production multi-tenant serving with >5 concurrent users. Use vLLM.
• Anyone needing AWQ-INT4 fit on 24 GB cards. Ollama can't load it; pick vLLM or ExLlamaV2.
• Workloads requiring tensor-parallel multi-GPU scaling. Ollama splits a model across GPUs but doesn't scale linearly; see /guides/running-local-ai-on-multiple-gpus-2026.

Related

• Stacks: /stacks/local-coding-agent, /stacks/offline-rag-workstation
• System guides: /systems/quantization-formats, /setup
• Hardware: RTX 4090, Apple M3 Ultra, RTX 3090
• Errors: /errors/wsl2-gpu-not-detected, /errors/ollama-bind-port-conflict, /errors/rocm-device-not-found

Download from ollama.com/download — macOS, Linux, and Windows (preview) installers available. On Linux: curl -fsSL https://ollama.com/install.sh | sh. After install, Ollama runs as a background service. Pull a model: ollama pull llama3.2. Run interactively: ollama run llama3.2. The service exposes an OpenAI-compatible API at http://localhost:11434/v1/chat/completions. Verify: curl http://localhost:11434/v1/chat/completions -H "Content-Type: application/json" -d '{"model":"llama3.2","messages":[{"role":"user","content":"Hello"}]}'. Ollama handles model download, quantization selection, VRAM/system-memory management, and GPU layer offloading automatically — no manual GGUF selection or config flags needed. First run pulls a ~2 GB Q4 model and starts serving in 2–10 minutes depending on connection speed. Time-to-first-response from zero: ~3 minutes for a 3B model on a typical broadband connection. List models with ollama list, remove with ollama rm <name>. Customize via Modelfile: ollama create my-model -f Modelfile.

Best for: single-user local LLM with zero-config setup, developers who need an OpenAI-compatible local API in under 5 minutes, prototyping and experimentation with different models without manual quantization management, local RAG applications that talk to Ollama's embeddings endpoint, CI/CD pipelines that need a disposable LLM backend with minimal setup, any scenario where "just make it work on my laptop" is the primary requirement. Not suited for: multi-user production serving (use vLLM), latency-sensitive production with concurrency (Ollama queues requests sequentially), fine-tuning operations (Ollama is inference-only), Windows-native deployments without WSL2 (Windows preview is improving but not production-grade), scenarios needing fine-grained scheduler or memory control.

Use Ollama as the default local LLM runtime for single-user scenarios — it has the lowest setup friction of any inference tool and auto-selects quantization appropriate for your hardware. Switch to LM Studio when you want a GUI for model discovery and a visual chat interface alongside the API. Switch to llama.cpp when you need fine-grained control over GPU layer count, context length, flash attention, or speculative decoding parameters — Ollama abstracts these decisions away, which is fast but removes tuning knobs. Move to vLLM when you need multi-tenant production serving with concurrency >1 and the continuous batching throughput that Ollama doesn't provide. Switch to MLX-LM on Apple Silicon when you want native Swift/Apple-ecosystem integration and the best Metal performance; Ollama on macOS uses Metal via llama.cpp but MLX-LM occasionally benchmarks 5–15% faster on throughput.

Problem: Error: pull model manifest: file does not exist. Fix: Ollama's model library is versioned — the model name may have changed upstream. Check ollama.com/library for the current name (e.g. llama3.2:latest not llama3.2:3b). Use ollama pull llama3.2:latest explicitly. Problem: Server not responding on port 11434 after install. Fix: On macOS, Ollama runs as a menu bar app and starts the server automatically — check the menu bar icon. On Linux, the systemd service may be stopped: sudo systemctl start ollama. On Windows, Ollama runs as a system tray app. Problem: Model loads but produces gibberish output. Fix: You likely have the wrong template format for the model. Ollama auto-detects the chat template from the GGUF metadata but some custom GGUF files lack this. Create a Modelfile with FROM ./model.gguf and explicitly set TEMPLATE to the correct ChatML/Llama3/Mistral format, then ollama create fixed-model -f Modelfile.