Local AI tools

Personal AI agent with a local-first gateway architecture. Connects your local LLMs (Ollama, llama.cpp) to the messaging surfaces you already use — WhatsApp, Telegram, Slack, Discord, iMessage, and 20+ more. The runaway

Reference MCP server for fetching and converting web content. Pulls a URL, runs HTML through a readability extractor, returns markdown the model can chunk and reason over. The lightweight web-reader pair to Brave Search

AMD's open-source equivalent of NVIDIA CUDA. The stack behind AMD GPU inference in vLLM, PyTorch and llama.cpp's HIP build on Linux; llama.cpp and Ollama also offer a Vulkan path. Windows ROCm is improving as of 2026 but

Open-source embedding database for LLM applications. The default 'just install pip and start' vector store for prototypes, with first-party clients in Python and JS. SQLite-backed locally, distributed mode in cloud.

Anthropic's reference MCP server for filesystem access. Read, write, search, move, and list files inside a configured allowlist of directories. The canonical example for understanding how MCP tool exposure works in pract

Reference MCP server for local Git repository operations. Status, diff, log, show and branch listing, plus write tools such as add, commit, reset, checkout and merge, against a checked-out repo without round-tripping to

GitHub's first-party MCP server. Surfaces issues, pull requests, code search, file contents, repo metadata, Actions runs, and discussions through the protocol. Now maintained by GitHub itself rather than the original Ant

Reference MCP server that gives an agent a persistent knowledge graph — entities, relations, observations stored to disk and surfaced back across sessions. The simplest path to making an agent remember context between co

Long-term memory platform for AI agents. Sits above Graphiti as the application layer — sessions, facts, summaries, vector + graph hybrid retrieval. The 'memory backend you don't have to build' choice. Zep stopped mainta

HuggingFace's inference server, now in maintenance mode: the GitHub repo is archived and HuggingFace recommends vLLM, SGLang, llama.cpp or MLX going forward. Tight integration with the HF ecosystem.

PyTorch's official mobile / edge inference runtime. Compiles PyTorch models to a mobile-optimized format for Android (XNNPACK CPU, Vulkan GPU, Qualcomm, MediaTek and other accelerators) and iOS (XNNPACK CPU, Core ML). Th

Microsoft's cross-platform inference runtime for ONNX models. The reference path when you need a single runtime API that targets CUDA, DirectML, CoreML, OpenVINO and AMD MIGraphX through Execution Providers, with separat

Microsoft's ONNX Runtime packaged for Android and iOS apps, with NNAPI, CoreML and XNNPACK acceleration. The default mobile packages carry the full operator set; a custom build can cut them down to the operators a model

Intel's inference toolkit. The first-class path for Intel Arc GPUs, Intel NPUs (Lunar Lake / Meteor Lake), and CPU-optimized inference on x86. Pre-converted models, including INT8 variants, are published under the OpenVI

Distributed model serving on top of Ray. Lets you stitch vLLM / SGLang / custom runtimes into a multi-replica, multi-model deployment with autoscaling, traffic splitting, and pipeline composition. The orchestration layer

The CLI for the world's model hub. `hf download`, `hf upload`, and `hf models card` to fetch a model card.

Specialized transformer inference engine. The reference runtime for Whisper (faster-whisper), NLLB translation, and other encoder-decoder models. Out-of-the-box INT8 quantization with strong CPU performance.

Microsoft's DirectX 12 inference backend. In maintenance mode: Microsoft plans no new functionality and points Windows 11 version 24H2 and later to Windows ML. It gives GPU acceleration on DirectX 12-capable hardware fro

Anthropic's official desktop app for Claude. Native MCP server support means you can plug in local file access, GitHub, and custom tools. Distinct from the Claude Code CLI.

JetBrains' first-party AI for IntelliJ, PyCharm, WebStorm, etc. Multi-LLM backend (OpenAI, Anthropic, Gemini, local).

Terminal-based AI pair programmer. Run in your project directory, describe a change, it edits files and creates meaningful git commits. Works with any LLM — local Ollama, Anthropic, OpenAI, etc.

Anthropic's terminal-native coding agent. Deep MCP integration, agentic file editing, and a $20/mo Pro tier are the standout signals.

Coding agent for VS Code, JetBrains and the CLI, with about 5.9M VS Code Marketplace installs as of November 2026. Plan/Act mode, autonomous file edits with diff approval, terminal access. The leading open-source IDE age

Open-source coding agent for VS Code, JetBrains and the CLI. Acquired by Cursor and no longer actively maintained; 2.0.0 was the final release. Configurable autocomplete + chat + agent modes. Strong with local Ollama bac

Cognition's fully autonomous SWE agent. Cloud sessions plus a desktop app and a local CLI, longest task horizons. Plans run from free to $200/month for individuals; team plans have an $80/month minimum, with full seats a

Factory's autonomous SWE agent. Operates over GitHub PRs, Slack, Linear. Targets the long-running multi-file change workflow.

Open-source extensible AI agent now governed by the Agentic AI Foundation (AAIF) at the Linux Foundation. Started inside Block (formerly Square). 25+ provider support including Ollama, Ramalama, Docker Model Runner. Best

Open-source coding agent for VS Code, JetBrains and the CLI. Kilo reports 5M+ users, supports 500+ models, and charges zero markup over upstream API costs, with a 5% processing fee listed for credit purchases. The Kilo C

Agent memory framework that models memory like an operating system. Main context = RAM, archival storage = disk; the agent itself decides when to page. Originally MemGPT, now Letta. Model-agnostic (Anthropic, OpenAI, Oll

Open protocol for LLM clients to talk to external tools and data sources. The 'USB-C for AI' that became the default in 2026 — supported by Anthropic, OpenAI, and Google DeepMind, with more than 10,000 active public MCP

Drop-in memory layer for LLM agents. Open-source extraction is single-pass and ADD-only, with hybrid semantic, keyword and entity retrieval when the `nlp` extra is installed. Graph memory was removed from the open-source

OpenAI's 2025 coding agent (the new Codex, distinct from the deprecated 2021 model). Cloud task-runner pattern: hand it a multi-step task, it works in a sandbox and returns a PR.

Open-source CLI client for the new Codex agent. Local CLI that orchestrates cloud Codex models against your file tree.

Open-source coding agent built by Anomaly, with terminal, desktop and mobile interfaces. BYO LLM, MCP-compatible. A Claude-Code-style workflow without the Anthropic lock-in.

AI-driven development agent that completes engineering tasks end-to-end — branches, code, PRs. Release 1.5.0 added a planning agent, and v1.17.0 enabled Plan Mode on self-hosted and local backends. Local-LLM-friendly via

Inflection AI's consumer assistant — voice-first, conversational, designed for personal use rather than coding. Powered by Inflection AI's stack of custom fine-tuned models.

Replit's full-stack scaffolder agent, launched in September 2025 and followed by Agent 4 in March 2026. Goes from prompt to deployed app on Replit's hosted runtime.

Open-source AI dev-team extension for VS Code (1.55M installs, 23.8k GitHub stars). **Discontinued: all Roo Code products — Extension, Cloud, and Router — shut down on May 15, 2026** with refunds for unused balances. The

Sourcegraph's AI assistant, available only on Sourcegraph Enterprise since Cody Free and Pro ended on July 23, 2025. Strong at large-codebase context retrieval thanks to the underlying Sourcegraph index.

YAML-config fine-tuning framework. Reference toolkit for the open fine-tuning community (Hermes, Dolphin, etc. all use it).

Fine-tuning framework. The maker states 2x faster training with 70% less VRAM and no accuracy loss ([Unsloth README](https://github.com/unslothai/unsloth)). The open-source package supports multi-GPU through Accelerate,

Document-oriented LLM frontend with workspaces. Connects to Ollama, LM Studio, OpenAI, Anthropic, etc. Strong document RAG.

The original Stable Diffusion frontend. Less actively developed in 2026 than ComfyUI but still has the cleanest UX for simple gen.

Node-graph image-generation UI. Standard for Stable Diffusion and Flux workflows. Endlessly customizable.

One of the original local-LLM apps from Nomic. Privacy-focused, runs on CPU, decent model library. The last GitHub release is v3.10.0 from February 2025, and the repo's last push was in May 2025.

Open-source desktop ChatGPT alternative. Privacy-first, runs offline, supports Hugging Face import.

Cross-platform desktop client supporting local and cloud models in one window. Strong on knowledge-stack RAG.

Character-driven LLM frontend originally for role-play; widely used for any persona-driven workflow. Supports OpenAI, KoboldAI, llama.cpp, Ollama, Aphrodite, oobabooga endpoints. Rich sampling controls, character cards,

The 'AUTOMATIC1111 of LLMs'. Kitchen-sink Gradio UI with multi-backend support and a big extension ecosystem.

Single-file llama.cpp distribution focused on roleplay and creative writing. Bundles a web UI, image gen, and the Kobold API.

Open-source ChatGPT clone with multi-provider support (OpenAI, Anthropic, local LLMs via OpenAI-compatible APIs). A popular self-hosted ChatGPT-shaped frontend. Strong multi-user + RAG + plugin support; pairs well with v

Polished desktop GUI for local LLMs. Built-in HuggingFace search, OpenAI-compatible local server, side-by-side conversations.

Self-hosted ChatGPT-style web frontend. Pairs with Ollama or any OpenAI-compatible backend. Multi-user, RAG built in, fast.

AI-native IDE from Anysphere, which SpaceX acquired in August 2026. Forks VS Code with Cursor Tab inline completion, agentic chat, and Cloud Agents. Best 'flow' for inline completion in 2026.

GitHub's incumbent AI assistant. VS Code, JetBrains, Neovim integrations. Lost some inline-completion mindshare to Cursor and agentic mindshare to Claude Code, but still the easiest enterprise rollout via GitHub.

AI IDE from Cognition, renamed from Windsurf to Devin Desktop in June 2026. The company behind Windsurf was earlier named Codeium. Devin Local agent, Tab completions, and a free plan with a light agent quota.

High-performance native editor from the Atom team, with built-in AI panel and inline assistant. BYO API key for any provider.

Python/JS framework for chains, agents, and RAG. Batteries-included but heavyweight; many graduate to LangGraph or DIY.

Python/JS framework focused on RAG and document indexing. Cleaner than LangChain for retrieval-heavy use cases.

A coding agent optimized for low-cost models. It is a fork of OpenAI's Codex that emulates agent harnesses suited to those models, and it runs commands inside native sandboxing on macOS, Linux and Windows. The original P

Open-source LLM tracing + evaluation. OpenInference standard for traces; runs locally with one pip install. The OSS-first pick for teams that want LangSmith-shaped functionality without vendor lock-in.

Browser-style app launcher for AI tools. One-click installs of ComfyUI, oobabooga, RVC, and many other AI apps.

Inference engine based on vLLM, formerly named Aphrodite Engine and now named Sonar. Adds samplers (smoothing factor, dynatemp, mirostat, DRY, XTC) that mainline vLLM doesn't ship. Same continuous-batching architecture.

Archived by its developer, with development continuing on ExLlamaV3. Hand-optimized inference for EXL2-quantized models. Fastest single-GPU runtime for the EXL2 quant format on Ada/Hopper hardware. Lower-level than llama

The bedrock of local LLM inference. Most other tools wrap or embed it. Maximum control, maximum platform support, sharpest learning curve.

The default first-pull tool for local AI. One-line model installs (`ollama run llama3.1`), an OpenAI-compatible HTTP API, good defaults out of the box. Built on llama.cpp and GGML, with an MLX engine on Apple silicon.

Apple's Metal-native ML framework's LLM runner.

TVM-based LLM compilation framework. Compiles models for any GPU with a Vulkan / Metal / WebGPU / CUDA backend. A cross-platform on-device LLM runtime — runs Llama, Phi, Gemma, Qwen on phones, browsers, and laptops witho

Apple's Swift bindings for MLX. The native iOS / iPadOS path for on-device LLM inference. Apple-published example apps such as LLMEval download an LLM from Hugging Face and run it on iOS and macOS. LLM implementations no

Intel's PyTorch extension for low-bit LLM inference on Intel GPUs / CPUs / NPUs. Archived: Intel will not maintain it, no longer accepts patches, and says it has known security issues. Its last stable release, v2.2.0, is

Python bindings for llama.cpp with an OpenAI-compatible HTTP server. The fastest path from `pip install` to a working local-LLM endpoint. Ships pre-built wheels with optional CUDA / Metal / ROCm / Vulkan support.

Mozilla's single-binary llama.cpp distribution. Download one file, run on most operating systems with no installation.

Qualcomm's official on-device-AI compiler + model zoo for Snapdragon NPU targets. Pre-quantized model variants for Llama, Phi, Gemma, Qwen running on Hexagon NPU. The reference path for Android NPU acceleration in 2025-2

Personal AI cluster software. Auto-discovers Apple Silicon devices on a LAN and shards a model across them via pipeline + tensor parallelism on top of MLX. The [exo README](https://github.com/exo-explore/exo) says RDMA o

MCP server wrapping Firecrawl — a managed crawler that handles JavaScript rendering, anti-bot evasion, and large-site map+scrape jobs at scale. The pragmatic upgrade from mcp-server-fetch when an agent needs to crawl tho

Temporal graph memory framework. Builds a bi-temporal knowledge graph from agent conversations, tracking when each fact was learned and when it was true. Powers Zep's hosted offering.

Decentralized peer-to-peer AI inference network. Its GitHub repo description claims 2.7M+ downloads and its README claims over 2 million agents worldwide. Three-tier model routing (local registry → DHT → gossip broadcast

Embedded vector + columnar database. Lance file format reads serverless from S3/local disk; no separate process to run. The pick for embedded apps and notebook workflows.

OpenAI-API-compatible drop-in for self-hosted inference, with a multi-backend twist: the same endpoint can serve LLMs (llama.cpp / vLLM under the hood), embeddings, image gen (stable-diffusion.cpp), audio (whisper.cpp),

Brave's official MCP server wrapping the Brave Search API; it replaced the reference server, which is now archived. Privacy-respecting alternative to Google/Bing endpoints — Brave does not maintain a personal-history-lin

Microsoft's MCP server that drives a real browser via Playwright — Chromium, Firefox, and WebKit. Ships 25 tools by default that operate against the page's accessibility tree rather than pixel coordinates (its README lis

Reference MCP server that exposes a Postgres database as a query surface. Read-only is its only mode: each query runs inside a READ ONLY transaction. It is archived: the final npm release, 0.6.2, is deprecated as "Archiv

Reference MCP server that gives an agent a structured scratchpad for multi-step reasoning. Each call records a numbered thought with revision and branching support — the agent can backtrack, fork, and consolidate plans w

Distributed vector database designed for billion-scale workloads. Compute-storage separation, GPU-accelerated index builds, multi-tenant from the ground up. The pick when you've outgrown Qdrant single-node.

BitTorrent-style decentralized LLM inference. Splits a model into transformer-block shards distributed across volunteer hosts on the public internet — one client runs the input/output layers locally and streams activatio

Vector database written in Rust. Strong filtering (payload-based pre-filter), HNSW index with quantization variants, gRPC + REST APIs. The performance pick when you cross 10M vectors.

Vector search inside the same Redis you already run. HNSW + flat indices, hybrid filtering with FT.SEARCH. The pragmatic pick when you don't want to add another service to ops.

OpenAI-API frontend for ExLlamaV3. Wraps the Exllama inference engine in a clean HTTP API, adds streaming, batching, and OAI-compatible chat templates. A front-of-house when you've already committed to the EXL3 quant for

Structured generation language + runtime for LLM programs. RadixAttention reuses KV cache across prompts with shared prefixes — significant throughput wins for agent workloads where many tool calls share system prompts.

NVIDIA's first-party LLM inference library, architected on PyTorch. PyTorch is the default backend, so HuggingFace checkpoints load directly with no engine-build step, and the current 1.4 release candidates remove the ol

High-throughput inference engine with PagedAttention, continuous batching, and tensor + pipeline parallelism. The reference deployment runtime when you've outgrown llama.cpp / Ollama for production serving. Began in UC B

LangChain's observability + evaluation platform. Trace agent runs, run evaluators against benchmark suites, version prompts. The dominant trace+eval tool for the LangChain/LangGraph ecosystem.

Neo4j's official GraphRAG toolkit — Python library + reference patterns for building retrieval-augmented generation against a knowledge graph. The mature pick for enterprises already running Neo4j.

Vector database with built-in modules for embedding, generative search, and reranking. Schema-first design appeals to teams used to traditional databases. Generative-search module pairs with local Ollama models out of th