LlamaIndex

What it is and how it works

LlamaIndex is a data framework built specifically around retrieval-augmented generation (RAG): getting your own documents, databases, and APIs into a shape an LLM can query accurately. Where a general-purpose agent framework tries to be everything — chains, agents, memory, tool-calling, orchestration graphs — LlamaIndex starts from the opposite end of the problem: how do you take a pile of unstructured PDFs, Notion pages, SQL tables, or Slack exports and turn them into something an LLM can retrieve from with high recall and low noise. That "RAG-first" framing, called out directly in its own description, is the core design decision that shapes everything else about the project.

The architecture is organized around a handful of primitives that map cleanly onto the RAG pipeline: data connectors (loaders that pull in content from a source), document/node objects (chunked representations of that content with metadata attached), indexes (vector store indexes, list indexes, tree indexes, keyword table indexes, and knowledge-graph indexes, each optimized for a different retrieval pattern), and query engines (the layer that turns a natural-language question into a retrieval plan, fetches nodes, and synthesizes an answer). On top of these sits a retriever abstraction that lets you swap in different retrieval strategies — dense vector search, sparse/BM25, hybrid, or multi-step retrieval — without rewriting the rest of the pipeline. LlamaIndex is deliberately storage-agnostic: it doesn't ship its own vector database, it integrates with the usual suspects (Chroma, Weaviate, Qdrant, Pinecone, pgvector, Milvus, Elasticsearch, and others) through a common interface, plus it supports being paired with any LLM provider, local or hosted, through its own abstraction layer.

More recently the project has grown an agent layer on top of the retrieval core — workflows (an event-driven orchestration primitive), tool-calling agents, and multi-agent patterns — which puts it in more direct competition with LangChain/LangGraph than it was in its earlier "just an indexing library" days. But the agent tooling is still visibly built on top of the RAG foundation rather than being the framework's original reason for existing, and that lineage shows in how much more polished the ingestion/indexing/retrieval APIs feel compared to the newer orchestration layer.

Deployment patterns

For a solo developer or small team, the typical LlamaIndex deployment is a Python (or TypeScript, via LlamaIndex.TS) service embedded in an existing application: a script or FastAPI endpoint that loads documents through a SimpleDirectoryReader or a specific connector, builds a VectorStoreIndex against a local or hosted vector store, and exposes a query_engine.query() call behind an API route. On a laptop, this often runs entirely locally — local embedding model (e.g., via Ollama or a sentence-transformers model), local vector store (Chroma or a local FAISS index), and either a local LLM through an OpenAI-compatible endpoint (llama.cpp server, Ollama, vLLM) or a hosted API key. Because LlamaIndex treats the LLM and embedding model as swappable providers, running the whole pipeline offline against local models is a first-class, well-supported path, not a workaround.

In a homelab or small-team server setting, the pattern shifts toward a persisted index: documents are ingested once (or on a schedule via an ingestion pipeline with caching/dedup), embeddings are written to a standalone vector database running as its own service (Qdrant or Weaviate in Docker are common choices), and the LlamaIndex application layer becomes a thin, stateless query service that can be scaled horizontally since the heavy state lives in the vector store. Re-ingestion on document change is handled through LlamaIndex's ingestion pipeline abstractions, which support incremental updates rather than full rebuilds.

At team/production scale, LlamaIndex is usually one component in a larger stack — the indexing and retrieval layer sitting behind an application backend, often paired with LlamaCloud (the company's managed parsing/ingestion service) for document parsing quality on messy PDFs, plus observability tooling (LlamaTrace, or generic LLM tracing like Phoenix/Arize) bolted on for debugging retrieval quality in production. Teams running fully local/air-gapped stacks skip LlamaCloud and rely on open-source parsers (unstructured.io, PyMuPDF) instead.

How it compares

The most obvious comparison is LangChain, and it's a genuinely close one: both are Python/JS frameworks with broad LLM-provider and vector-store integrations, and both have grown agent orchestration layers over time. The practical difference operators report is that LlamaIndex's retrieval and indexing APIs are cleaner and more purpose-built — fewer layers of abstraction to get a solid RAG pipeline working — while LangChain has the larger ecosystem, more third-party integrations, and a more mature (if more sprawling) agent/orchestration story via LangGraph. If the job is "build accurate retrieval over documents," LlamaIndex tends to get there with less boilerplate; if the job is "build a complex multi-tool agent with many integrations," LangChain's ecosystem breadth often wins.

Haystack (deepset) is the other direct RAG-framework peer, and it's arguably more enterprise/pipeline-oriented — its explicit DAG-based pipeline model is more rigid but more predictable in production than LlamaIndex's more implicit query-engine abstractions. Haystack also has a smaller community than either LlamaIndex or LangChain.

For teams that don't want a Python framework at all, tools like LangChain's simpler cousins or raw vector-DB SDKs (just calling Chroma/Qdrant directly plus manual prompt construction) remain a valid lower-abstraction alternative — more code to write, but no framework lock-in and easier to reason about when things go wrong. LlamaIndex earns its keep specifically when the ingestion/chunking/retrieval logic is complex enough (multiple document types, hybrid retrieval, re-ranking, query transformations) that hand-rolling it stops being worth the time.

Best use cases and honest limitations

LlamaIndex is a strong default for anyone building production RAG: internal knowledge-base search, document Q&A over PDFs/Notion/Confluence, customer-support retrieval bots, or codebase-aware assistants. The RAG-first design genuinely pays off here — the framework's opinions about chunking, node relationships, and retrieval strategies are well-tested defaults that save real engineering time versus building the same logic from scratch.

The honest limitation, echoed in its own cons, is ecosystem size: LangChain simply has more integrations, more Stack Overflow answers, more third-party tutorials, and a larger pool of community-maintained connectors, which matters when you hit an edge case. Teams needing a broad, mature agent-orchestration story with many pre-built tool integrations may still find LangChain (or purpose-built agent frameworks) a better fit than bending LlamaIndex's newer workflow layer to that purpose. It's also worth noting that LlamaIndex's most polished commercial offering, LlamaCloud, is a hosted parsing service — fully usable without it, but teams chasing the best PDF-parsing quality may find themselves nudged toward a non-local dependency if they don't want to self-host equivalent parsing tooling. For anyone whose problem isn't primarily "retrieve accurately from my own data" — e.g., pure conversational agents, workflow automation without a document corpus — a lighter-weight framework or no framework at all is often the better call than adopting LlamaIndex's abstractions.

Install via pip: pip install llama-index llama-index-embeddings-openai llama-index-llms-openai. Requires Python 3.10+. LlamaIndex is a data framework for LLM applications focused on ingestion, indexing, and retrieval over your own data. Quick start: from llama_index.core import VectorStoreIndex, SimpleDirectoryReader; documents = SimpleDirectoryReader("./data").load_data(); index = VectorStoreIndex.from_documents(documents); query_engine = index.as_query_engine(); response = query_engine.query("What is this document about?"). This creates an in-memory vector index from text files in ./data/ and queries it. For persistent storage: swap the default in-memory index for a vector store (ChromaDB, Pinecone, Qdrant, Weaviate) via their integration packages (pip install llama-index-vector-stores-chroma). Local models: pip install llama-index-llms-ollama llama-index-embeddings-huggingface then use Ollama or HuggingFaceEmbedding. LlamaIndex supports 160+ data connectors (PDF, SQL, Notion, Slack, GitHub, etc.) via llama-index-readers-* packages. First run: ~3 minutes for install, downloads model + indexes documents. Verify: the quick start example above returns an answer about your documents. Time-to-first-RAG: ~10 minutes.

Best for: RAG applications over private document collections (PDF reports, knowledge bases, manuals), data ingestion pipelines that parse and index heterogeneous document formats (160+ connectors), complex retrieval patterns (sub-question decomposition, recursive retrieval, hierarchical indexing), teams standardizing on "chat with your data" use cases, knowledge management and enterprise search built on LLM retrieval. Not suited for: general LLM application frameworks beyond data retrieval (use LangChain), simple Q&A over a few documents where direct embedding + cosine search suffices, applications where latency budget excludes the ingestion+retrieval pipeline overhead, non-Python environments (LlamaIndex has a TypeScript port but it trails the Python library), one-off queries where framework setup cost exceeds value.

Use LlamaIndex when your primary use case is RAG (retrieval-augmented generation) over your own data — document ingestion, chunking, embedding, indexing, and retrieval are LlamaIndex's core competency and the pipeline is more polished than LangChain's. Switch to LangChain when you need a broader application framework beyond RAG (chains, agents, tool orchestration) — LlamaIndex is data-centric, LangChain is application-centric. Use Haystack for an alternative RAG framework with a more opinionated pipeline API. Use direct vector DB + LLM integration for simpler RAG needs where a framework adds overhead. LlamaIndex's strengths: the ingestion pipeline (parsing → chunking → metadata extraction → indexing) handles PDFs, complex documents, and hierarchical data better than competitors; the query engine abstraction (router, sub-question, recursive) makes complex retrieval patterns straightforward. Its weakness: overlapping with LangChain's RAG features creates ecosystem confusion, and the package ecosystem (llama-index-*) has grown fragmented.

Problem: ModuleNotFoundError: No module named 'llama_index' after pip install. Fix: The package name is llama-index (with hyphen), not llama_index (underscore). The import uses underscore: from llama_index.core import .... Ensure llama-index-core is installed (it's a dependency of llama-index). For specific integrations, install separately: pip install llama-index-embeddings-huggingface etc. Problem: RAG returns "I don't have enough information" despite relevant documents. Fix: Check the chunking strategy — default chunk size (1024 tokens) may fragment key information across chunks. Reduce chunk overlap or increase chunk size: Settings.chunk_size = 2048; Settings.chunk_overlap = 200. Check the retrieval window — the default similarity_top_k=2 may be too low. Increase to 5–10. Use query_engine = index.as_query_engine(similarity_top_k=10). Problem: Document ingestion hangs on large PDFs. Fix: Some PDF readers (PyPDF2) are slow on scanned/image-heavy PDFs. Switch the reader: from llama_index.readers.file import PDFReader with parser=PyMuPDFReader(). For scanned PDFs, enable OCR: parser = PDFReader(ocr_languages=["eng"]). Split ingestion into batches with num_workers=4 for parallel processing.