Retrieval-Augmented Generation (RAG)

RAG is the pattern of retrieving relevant documents from a knowledge base and including them in the LLM's prompt so the model can ground its answer in those documents. The simplest RAG: embed your docs, embed the user's query, find the top-k nearest doc chunks by cosine similarity, prepend them to the prompt.

RAG addresses three problems: knowledge cutoff (new info the model wasn't trained on), private data (docs you can't retrain on), and hallucination (model making things up). It does NOT make the model more capable — a 7B model with RAG is still a 7B model on the reasoning side.

Key design decisions: chunk size (256-1024 tokens typically), embedding model (BGE, E5, or text-embedding-3 from OpenAI), vector store (Postgres pgvector, Qdrant, Weaviate, ChromaDB), retrieval quantity (top-3 to top-20). For local-only RAG, llama.cpp + a small embedding model + SQLite-based vector store gets you started in under 100 lines of code.

An operator needs a local assistant that answers questions about a 400-page internal engineering wiki, updated weekly, without retraining anything. They chunk the wiki into ~512-token sections, embed each chunk with a local BGE-small model, and store the vectors in a vector database like Qdrant running in Docker. At query time, the user's question gets embedded, the top-5 nearest chunks are retrieved by cosine similarity, and those chunks get prepended to the prompt sent to a locally-hosted Llama 3.1 8B via llama.cpp. When the wiki updates, they only re-embed the changed pages — no model retraining. The setup catches the two classic RAG failure modes early: chunks that split mid-sentence (fixed with overlap) and irrelevant retrievals when the query wording diverges from the doc wording (improved by using a better embedding model, not a bigger LLM).