KoboldCPP

Single-file llama.cpp distribution focused on roleplay and creative writing. Bundles a web UI, image gen, and the Kobold API.

Download the latest koboldcpp.exe (Windows) or the platform binary from github.com/LostRuins/koboldcpp/releases. On Linux/macOS, build from source: git clone https://github.com/LostRuins/koboldcpp && cd koboldcpp && make. KoboldCPP bundles llama.cpp as its inference backend and wraps it with a built-in web UI. Launch: ./koboldcpp --model models/Llama-3.2-3B-Instruct-Q4_K_M.gguf --port 5001. The web UI opens at http://localhost:5001. The API (KoboldAI-style + OpenAI-compatible) is at http://localhost:5001/api/v1/generate and http://localhost:5001/v1/chat/completions. Verify: curl http://localhost:5001/api/v1/generate -H "Content-Type: application/json" -d '{"prompt":"Hello"}' . KoboldCPP auto-offloads layers to GPU if CUDA or Vulkan is available. Context shifting (SmartContext) preserves conversation context across long chats by shifting the KV cache rather than truncating. Time-to-first-response: ~10 seconds after model load for a 7B GGUF. No Python needed — single binary.

Best for: creative writing and roleplay with world-info lore books and author's-note steering, Windows-first local LLM deployment without Docker or Python, users who want a self-contained binary with built-in UI and API, long-form storytelling with context-shifting that preserves narrative continuity beyond context window, SillyTavern and character-chat frontend integration via KoboldAI API, scenarios where the operator wants a complete experience (UI + API + prompt management) in one download. Not suited for: production multi-tenant serving (use vLLM), non-Windows developers who prefer CLI-native tools (use Ollama), maximum-throughput GPU inference (KoboldCPP inherits llama.cpp's throughput ceiling), fine-tuning, embedding generation.

Use KoboldCPP when you want a Windows-native, single-binary local LLM with a bundled web UI, roleplay-first features (world info, author's note, instruct mode), and the full llama.cpp model ecosystem. Switch to Ollama for CLI-first model management and auto-download — KoboldCPP requires you to source your own GGUF files. Use llama.cpp directly when you need the raw server without the UI and roleplay features. Use LM Studio for the best GUI model discovery and visual chat — KoboldCPP's UI is functional but web-based, not native. Use Aphrodite Engine on NVIDIA GPU for higher single-stream throughput. KoboldCPP uniquely bundles context-shifting, world-info management, and instruct-mode toggles that no other engine's built-in UI provides — it's the reference creative writing/roleplay engine.

Problem: SmartContext: failed to process context error on long conversations. Fix: SmartContext shifts the KV cache when the conversation exceeds context length. If it fails, disable it with --noshift and the system falls back to truncation (oldest messages dropped). Increase context with --contextsize 16384 to give more headroom before shifting is needed. Problem: GPU not detected on Windows with NVIDIA card. Fix: KoboldCPP uses CLBlast by default on Windows. For CUDA acceleration, download the koboldcpp_cuda.exe build variant from releases. For Vulkan (Intel/AMD), use the koboldcpp_rocm.exe build. The standard koboldcpp.exe is CPU-only. Problem: Generated text cuts off mid-sequence. Fix: Increase --maxlength from default 512. KoboldCPP's generation limit includes both input context and output tokens — a 2000-token context with 512 maxlength leaves -1488 budget for output, which causes early cutoff. Set --maxlength 8192 for long-form generation.