Agent Memory (Short/Long/Episodic)

Agent memory refers to the mechanisms an AI agent uses to store and recall information across interactions. Short-term memory holds context within a single session (like a conversation), typically limited by the model's context window (e.g., 4K-128K tokens). Long-term memory persists across sessions, often implemented via vector databases (e.g., Chroma, FAISS) that store embeddings of past interactions. Episodic memory records specific events or experiences, enabling the agent to recall past actions and outcomes. For operators, the key trade-off is between context window size (VRAM-bound) and external storage latency.

In practice, short-term memory is the model's context window—the tokens it can attend to at once. For example, a 32K context window can hold about 24,000 words. Long-term memory uses retrieval-augmented generation (RAG): past interactions are chunked, embedded, and stored in a vector database. When a new query arrives, the agent retrieves relevant chunks and injects them into the context. Episodic memory is a specialized form of long-term memory that stores structured records (e.g., 'user asked about X, I responded with Y, user was satisfied'). This allows the agent to learn from past mistakes or preferences. Operators can implement these using frameworks like LangChain or custom scripts with Ollama and Chroma. The main constraint is retrieval latency—embedding and searching take time, and the retrieved content consumes context window space.

An agent running on an RTX 4090 (24 GB VRAM) uses a 32K context window for short-term memory. For long-term memory, it stores conversation embeddings in a local Chroma database. When a user asks a follow-up question about a topic from yesterday, the agent retrieves the top-3 relevant chunks (~1K tokens) and prepends them to the current context. This consumes ~3% of the context window but adds ~50ms retrieval latency. Episodic memory might log that the user prefers concise answers, so the agent adjusts its system prompt accordingly.

In Ollama, an agent can be built by combining a model (e.g., llama3.1:8b) with a Python script using the chromadb library. The workflow: 1) On startup, load existing embeddings from disk. 2) For each user message, embed it and query Chroma for similar past interactions. 3) Prepend retrieved chunks to the prompt. 4) Run ollama run llama3.1:8b with the augmented prompt. 5) After the response, store the new interaction as an embedding. This runs entirely locally, with VRAM usage determined by the model size plus context overhead.