Letta (memory framework)

What it is and how it works

Letta (formerly MemGPT) is an agent framework built around a single, opinionated idea: agent memory should be managed the way an operating system manages memory, with a small, fast "main context" analogous to RAM and a much larger, slower "archival storage" analogous to disk. In a standard LLM chat loop, everything that has ever been said either stays crammed into the context window or gets silently truncated once the window fills up. Letta's approach instead treats the context window as a scarce, expensive resource that the agent itself is responsible for managing. The agent has explicit tool calls available to it — things like saving a fact to archival memory, searching archival memory, or editing its own persistent "core memory" blocks — and the LLM decides, turn by turn, what belongs in the limited main context versus what should be paged out to external storage and recalled later. This is the "self-editing memory" idea that made the original MemGPT paper notable: the memory hierarchy isn't bolted on by an external retrieval pipeline, it's a first-class capability the model actively directs.

Architecturally, a Letta agent is a stateful, server-side entity, not a stateless function call. Every agent has a persistent identity: its core memory blocks (things like persona and human/user info that stay pinned in context), its message history, and its archival and recall memory stores are all tracked server-side and survive across sessions. This is a meaningful departure from the typical "pass the whole conversation back in on every request" pattern used by most chat wrappers. Letta exposes this through a REST API and SDKs (Python and TypeScript/Node), so an application talks to an agent as a durable resource with an ID, similar to how you'd talk to a database-backed object rather than a chat completion endpoint.

The framework is model-agnostic by design — it sits above the model layer and can be pointed at Anthropic's API, OpenAI, Google Vertex, or local models served through Ollama or vLLM-style OpenAI-compatible endpoints. This matters for a self-hosted audience specifically because it means the memory-management architecture (the interesting, hard-to-replicate part) isn't tied to a specific hosted model — you can run the orchestration logic locally against a local model, at the cost of needing a model that's reliably good at structured tool-calling, since the entire memory system depends on the LLM correctly emitting memory-management function calls rather than just chatting.

Deployment patterns

The typical entry point is letta server, which spins up a local REST API server (backed by SQLite by default, with Postgres as the production-grade option) plus access to the Agent Development Environment (ADE), a web-based UI for inspecting an agent's live state — its core memory blocks, message queue, archival memory contents, and the raw tool calls it's making to manage its own memory. This inspectability is one of Letta's more distinctive operator-facing features: instead of a black-box conversation log, you can watch an agent decide to evict something from context or write a new memory block, which is genuinely useful for debugging why an agent "forgot" something or behaved inconsistently over a long-running session.

For a solo developer or homelab setup, the common pattern is docker run or pip install letta to get the server up, pointing the model config at either a hosted API key or a local Ollama/vLLM endpoint, and then building against the Python/TypeScript SDK or hitting the REST API directly from an application. Because agents are persistent server-side resources, this fits naturally into building actual products — customer-support bots, coding assistants, or personal assistants that need to remember user preferences across weeks of conversations — rather than one-off scripts. Postgres becomes relevant once you're running multiple agents concurrently or want durability guarantees beyond a local SQLite file; a small team server would run Letta server + Postgres behind the same network, with the ADE used collaboratively for debugging agent behavior. There's also a managed Letta Cloud option for teams that want the hosted-service tradeoff instead of operating the server themselves.

How it compares

The most direct comparison is Mem0, which occupies similar territory — persistent memory for LLM agents — but takes a much lighter-weight approach: a memory API you call from your own agent loop to store and retrieve facts, without Letta's OS-metaphor architecture or the requirement that the agent itself issue memory-management tool calls. Mem0 is faster to bolt onto an existing stateless agent; Letta requires more buy-in to its agent-loop model but gives the LLM actual agency over what it remembers and when, which tends to matter more for genuinely long-running, autonomous agents rather than a chatbot with a retrieval-augmented memory bolted on the side.

Against LangGraph or other general agent-orchestration frameworks, Letta is narrower in scope — it's not trying to be a general graph-based workflow engine, it's specifically a memory-and-state layer with an agent server around it. You could use LangGraph to orchestrate an agent that itself talks to a Letta-managed agent for memory, or you could pick one or the other depending on whether your hard problem is multi-step tool orchestration (LangGraph's strength) or long-horizon memory persistence (Letta's strength). Compared to simply using a vector database (Chroma, Qdrant) directly for RAG-style memory, Letta is a much higher-level abstraction — you get agent identity, a server, and an inspection UI, at the cost of adopting its full agent-loop paradigm rather than just doing similarity search yourself.

Best use cases and honest limitations

Letta is a strong fit for anyone building an agent that needs to persist meaningfully across sessions — a personal assistant that should remember your preferences three weeks later, a support agent that needs continuity across a long customer relationship, or research into long-horizon autonomous agents where the OS-memory metaphor is the actual object of study. The model-agnostic design pairs well with local Ollama deployments for operators who want to keep everything on-prem, and the Apache 2.0 license means there's no licensing friction for commercial use. The ADE is a genuine differentiator for debugging — most memory frameworks give you no visibility into why an agent behaved the way it did.

The honest tradeoffs: this is not a drop-in memory API. Getting good results depends on the underlying model reliably emitting correct tool calls for memory management, which pushes you toward stronger models and away from smaller local models that struggle with consistent structured output. The learning curve is real — you're adopting Letta's agent-loop paradigm, not just calling a remember() function — and for stateless, one-shot agent tasks (a single Q&A call with no need for continuity) it's meaningfully more machinery than the problem warrants. Teams that just need "remember these facts about the user" without the full agent-server architecture will likely find Mem0 or a plain vector store faster to integrate. Letta earns its complexity specifically when the agent's long-term behavior and self-directed memory management are the point, not an afterthought.

Install via pip: pip install letta. Requires Python 3.10+. Letta (formerly MemGPT) is a framework for building stateful LLM agents with persistent memory and tool use. Start Letta server: letta server. This starts the REST API at http://localhost:8283. Create an agent: letta create-agent --name my-agent --model gpt-4o. For local models: letta create-agent --name local-agent --model llama3.2 --llm-endpoint http://localhost:11434/v1 --llm-endpoint-type openai. The agent persists its memory (conversation history, core memories, archival memories) to a SQLite database. Chat via CLI: letta chat --agent my-agent. The Letta SDK provides programmatic access: from letta import Letta; client = Letta(base_url="http://localhost:8283"); agent = client.agents.get("agent-id"); response = client.agents.messages.create(agent_id=agent.id, messages=[{"role": "user", "content": "Hello"}]). First run: letta server auto-creates the SQLite DB, ~5 seconds start time. Time-to-first-agent: ~30 seconds including model prompt. Verify: letta chat --agent my-agent and send a message — the agent responds and persists memory to SQLite.

Best for: persistent AI companions that remember users and context across sessions, customer support agents that accumulate knowledge about accounts over time, personal AI assistants that grow their understanding of the user, research agents that maintain a growing knowledge graph from literature scanning, any application where "the agent should remember what we talked about yesterday" is a requirement. Not suited for: stateless single-turn Q&A (use direct LLM API calls), RAG over static document collections (use LlamaIndex), applications where memory-as-a-service is preferred over agent framework (use Mem0 or Zep), latency-sensitive real-time systems (memory paging adds 1–3 seconds per archival memory access), environments where SQLite doesn't meet persistence requirements (use the Postgres backend).

Use Letta when you need LLM agents with long-term persistent memory — an agent that remembers conversations across days, maintains a growing knowledge base, and self-edits its own memory. Letta's virtual context management (OS-inspired paging of memories between context window and persistent storage) is unique among open-source agent frameworks. Switch to Mem0 when you want memory as an API layer for any LLM application rather than a full agent framework — Mem0 is a memory service, Letta is a memory-native agent platform. Use LangChain agents when you need broader tool ecosystem integration without the memory primitives. Use CrewAI or AutoGen for multi-agent orchestration. Letta's strength: the memory architecture — it treats LLM context as an OS treats RAM and uses SQLite/Postgres as "disk" for paging memories in and out. Its weakness: heavier than simpler memory solutions (Mem0, Zep) and the full agent framework adds complexity when you only need memory.

Problem: letta server fails with "address already in use" on port 8283. Fix: Change the port: letta server --port 8284. If using the Letta client, specify: client = Letta(base_url="http://localhost:8284"). The REST API and admin UI both default to 8283. Problem: Agent memory doesn't persist between sessions with local models. Fix: Letta's memory management requires the model to respond correctly to function-calling prompts (for memory read/write/edit tools). Smaller local models may not implement tool calling reliably. The memory tools (core_memory_append, archival_memory_insert, core_memory_replace) are embedded in the system prompt as function definitions — if the model doesn't invoke them, memory doesn't update. Test with a tool-calling capable model (Llama 3.1 8B function-calling variant, Mistral 7B v0.3, or Qwen 2.5 7B). Problem: letta chat exits with "Agent not found." Fix: Agent state is stored in the SQLite database (~/.letta/letta.db by default). If you change the database or the server was reset, agents are lost. Run letta list-agents to see available agents. The server instance manages a single database — running multiple server instances with different DB paths isolates agents.