Vision
colpali
image rag
document image retrieval

Visual RAG

Retrieval-augmented generation over document images directly — no OCR pre-processing step. ColPali / ColQwen-style models embed page images for retrieval.

Capability notes

Visual RAG (Retrieval-Augmented Generation) retrieves documents by their visual appearance, not extracted text. The breakthrough model is ColPali (2024, by the ColBERT team) — it embeds entire document page images directly into a late-interaction retrieval space, skipping OCR entirely. ColQwen extends the approach using Qwen2-VL as the vision backbone. This paradigm shift means documents with complex layouts, handwritten text, diagrams, and non-Latin scripts are retrievable without pre-processing. ColPali on ViDoRe benchmark (document retrieval): NDCG@5 of 81.2 — outperforms traditional OCR → text embedding pipelines by 12–18 points. The gain comes from preserving visual signals lost during OCR: font weight indicates headings, spatial position conveys hierarchy, figures and charts contain retrievable information, document templates provide genre signals. A tax form and a scientific paper are visually distinguishable before a single word is read. How it works: ColPali encodes each page image into a multi-vector representation — typically 256–1024 token-level embeddings per page. At query time, the user's text query is encoded into a matching multi-vector representation. Late interaction (ColBERT-style MaxSim) computes relevance between query vectors and page vectors — a fine-grained matching that captures "the query word 'deduction' matches this specific table cell on the page." Storage: ~1024 vectors × 128 dimensions × 4 bytes = 512 KB per page. 1M pages = 500 GB vector storage. Multi-vector retrieval is the enabling technique. Traditional retrieval encodes each page as a single vector — cheap to store (1K–4K dimensions), fast to search, but lossy (one vector for an entire page). Multi-vector retrieval stores hundreds of vectors per page — 50–100× more storage, slower search, but preserves token-level information. ColPali's accuracy gain (12–18 NDCG points) justifies the storage and latency cost for document retrieval where finding the right page matters more than retrieval speed. For recommended models, see [Llama 3.3 70B](/models/llama-3-3-70b). Deploy locally with [Ollama](/tools/ollama) on [RTX 4090](/hardware/rtx-4090).

If you just want to try this

Lowest-friction path to a working setup.

Start with ColPali via the [Hugging Face Transformers](/tools/transformers) library. Install: ```bash pip install transformers torch pillow ``` Load ColPali and embed a document: ```python from transformers import ColPaliForRetrieval, ColPaliProcessor import torch model = ColPaliForRetrieval.from_pretrained( "vidore/colpali-v1.2", torch_dtype=torch.bfloat16 ).to("cuda") processor = ColPaliProcessor.from_pretrained("vidore/colpali-v1.2") # Embed a document page from PIL import Image image = Image.open("document_page.jpg") inputs = processor(images=[image], return_tensors="pt").to("cuda") with torch.no_grad(): embeddings = model(**inputs) # shape: [1, N_patches, dim] print(f"Embedding shape: {embeddings.shape}") # ~1030 patches × 128 dims ``` Store embeddings in a vector database. For small collections (<10K pages), use [Qdrant](/tools/qdrant) with multi-vector support. Install: `pip install qdrant-client`. Create a collection with multi-vector configuration, index page embeddings as numpy arrays. For interactive search on your own documents, use [Byaldi](https://github.com/AnswerDotAI/byaldi) (Answer.AI) — a wrapper around ColPali/ColQwen with built-in indexing and search UI. Install: `pip install byaldi`. Index a directory of PDFs: `byaldi index --model colpali my_index/ pdfs/`. Search: `byaldi search my_index/ "query text"`. Byaldi handles PDF→image conversion, embedding, indexing, and retrieval — simplest path to working Visual RAG in under 10 minutes. ColPali v1.2 (PaliGemma-based, 3B params) runs on [RTX 3060 12GB](/hardware/rtx-3060-12gb) embedding ~2–5 pages/second. For larger collections, use ColQwen2-VL 7B which embeds at 1–3 pages/second with higher accuracy on multilingual documents.

For production deployment

Operator-grade recommendation.

Production Visual RAG requires careful index design and retrieval latency optimization. **Indexing pipeline.** PDF → pdf2image (200 DPI) → ColPali/ColQwen embedder → multi-vector index. Processing rate: 50–200 pages/minute on [L40S](/hardware/nvidia-l40s) with ColPali. For 1M page corpora: 3.5–14 days of GPU time on single L40S — plan for parallel indexing across 4–8 GPUs (1–2 days). Store raw page images alongside embeddings for retrieval display — adds 50–200 KB/page (JPEG compressed), 50–200 GB for 1M pages. Total index: 500 GB vectors + 100 GB images = 600 GB for 1M pages. **Retrieval latency.** Multi-vector search is computationally heavier than single-vector: ColBERT-style MaxSim computes pairwise similarity between query vectors (32 tokens) and document vectors (~1030 patches) — 32 × 1030 = 33K similarity operations per document. Brute-force search over 1M documents: 33B operations at ~5ms per query on [A100 80GB](/hardware/nvidia-a100-80gb-sxm). For production, use approximate search: FAISS IVF-PQ compresses multi-vectors and partitions index — reduces latency to 50–200ms at 90–95% recall. [Qdrant](/tools/qdrant) with multi-vector binary quantization achieves 100–300ms latency at 85–90% recall. Acceptable for interactive search; optimize further for sub-100ms requirements. **Two-stage retrieval for cost optimization.** Stage 1: single-vector retrieval using ColPali's CLS token embedding (fast, cheap — 5–20ms over 1M pages). Returns top-1000 candidate pages. Stage 2: multi-vector reranking with ColPali full embeddings on top-1000 (50–200ms). End-to-end: 55–220ms. Accuracy: within 2–5% of full multi-vector search. This architecture recovers 95% of ColPali's accuracy at 5–10% of the compute cost. **Visual RAG for Q&A.** After retrieval, feed retrieved page images + user query to VLM for answer generation. Qwen2.5-VL 72B or InternVL2-76B on [A100 80GB](/hardware/nvidia-a100-80gb-sxm): generate answer from 3–5 retrieved page images in 2–5 seconds. Total pipeline: 100–300ms retrieval + 2–5 seconds generation = 2.5–5.3 seconds end-to-end. For faster responses (sub-second), route simple queries to text-based RAG and visual queries to Visual RAG. **When to use Visual RAG vs text RAG.** Visual RAG wins when: documents are scanned/handwritten (OCR quality <95%), documents contain important non-textual content (charts, diagrams, stamps, signatures), query domain requires visual template matching (legal forms, invoices, government documents). Text RAG wins when: documents are born-digital (OCR >99%), query is purely textual/semantic ("find all documents about renewable energy policy"), storage/latency budget is tight (<$1K/month total infrastructure).

What breaks

Failure modes operators see in the wild.

**Late interaction collapse on long documents.** Symptom: 50-page PDF — query matches a specific sentence on page 37, but ColPali ranks page 1 (cover page) higher because MaxSim aggregates patch scores and cover pages have diverse visual features that match many query terms by chance. Cause: MaxSim is a sum-of-max-similarities — pages with many distinct visual regions (cover pages, table-of-contents) accumulate high scores even without semantic relevance. Mitigation: apply length normalization (divide MaxSim score by number of patches), use query-side filtering (CLS token pre-filter removes cover/TOC pages before multi-vector comparison), apply IDF weighting to down-weight common visual patterns. **Page segmentation granularity mismatch.** Symptom: document has 3 distinct sections on one page (text block, sidebar, footer ad) — ColPali retrieves the page for matching "sidebar" content but the user wanted "text block" content. Cause: ColPali embeds entire page as one retrieval unit — cannot distinguish within-page regions. Mitigation: pre-segment pages into logical regions (layout parser), embed each region independently. Tradeoff: 3–8× more embeddings, 3–8× larger index, but precision improves 8–15 points on within-page retrieval tasks. **Multi-page context loss.** Symptom: user asks "what is the total revenue in Q3?" — answer requires data from page 4 (income statement) and page 15 (segment breakdown). ColPali retrieves page 4 but not page 15 because query terms match the income statement more strongly. Cause: retrieval treats each page independently — no understanding of cross-page document structure. Mitigation: after retrieving top-k pages, expand retrieval with "neighboring pages" (pages adjacent to top result within same document) and "document siblings" (other pages from same document that were top-scoring). Post-retrieval, prompt VLM with a document map (table of contents showing what each page contains).

Hardware guidance

**Hobbyist tier ($1,500–2,500).** [RTX 4090](/hardware/rtx-4090) at 24 GB: ColPali v1.2 (3B params) indexing at 2–5 pages/second, query embedding at 5–10 queries/second. 24 GB handles embedding batch=4 for faster indexing. Indexes ~7K–18K pages/hour — a personal document library (10K–50K pages) takes 1–7 hours to index. [RTX 5090](/hardware/rtx-5090) at 32 GB: ColQwen2-VL 7B at 1–3 pages/second, better multilingual accuracy. Fits larger batch sizes for indexing throughput. **SMB tier ($6,000–15,000).** [RTX 6000 Ada](/hardware/rtx-6000-ada) at 48 GB: ColPali at 5–10 pages/second batch=8. Indexes ~18K–36K pages/hour — handles departmental collections (500K–2M pages) in 2–7 days single GPU. [L40S](/hardware/nvidia-l40s) at 48 GB: identical indexing performance, 24/7 retrieval serving. **Enterprise tier ($25,000+).** [A100 80GB SXM](/hardware/nvidia-a100-80gb-sxm): ColPali batch=16 at 8–15 pages/second. ColQwen2-VL 7B at 3–6 pages/second. Parallel indexing across 4–8 A100s: 1M pages in 8–24 hours. Retrieval: multi-vector search on GPU with FAISS-GPU at 50–200ms per query over 1M pages. [H200](/hardware/nvidia-h200) at 141 GB: fits largest embedding models and full indices in GPU memory for sub-50ms retrieval. [AMD MI300X](/hardware/amd-mi300x) at 192 GB: can hold multi-million page indices entirely in GPU HBM — eliminates CPU↔GPU transfer latency during retrieval. **Vector storage hardware.** Multi-vector indices at scale (10M+ pages) exceed GPU memory. Storage tier: NVMe SSD array (RAID 0, 4× 4 TB = 16 TB) for index storage at 3–7 GB/s read bandwidth. Compute tier: GPU for embedding + retrieval. Retrieval: load relevant index shard to GPU for query, 50–200ms. This disaggregated architecture scales horizontally — add GPU nodes for embedding throughput, add SSD nodes for index capacity independently. **Apple Silicon.** [Mac Studio M3 Ultra](/hardware/mac-studio-m3-ultra) at 128 GB+ unified memory: ColPali at 3–8 pages/second. Index stays in unified memory — no GPU↔CPU transfer for retrieval, sub-100ms queries. Best developer workstation for Visual RAG prototyping and small-scale (<100K pages) production.

Runtime guidance

**ColPali vs ColQwen vs traditional RAG — retrieval architecture decisions.** [ColPali](https://github.com/illuin-tech/colpali) is the reference implementation. PaliGemma-3B backbone, Apache 2.0 license. Embedding dimension: 128. Patches per page: 1030 (at 448×448 input). Produces multi-vector embeddings compatible with any vector DB supporting multi-vector or ColBERT-style late interaction. FastEmbed integration: `pip install fastembed; from fastembed import ColPaliEmbedding` for simplified Python API. Best for: English-dominant document collections, clean documents, rapid prototyping. [ColQwen2-VL](https://github.com/illuin-tech/colqwen) extends the ColPali approach with Qwen2-VL backbone (2B or 7B). Better multilingual retrieval (trained on diverse scripts), stronger on handwritten and degraded documents. Embedding dimension: 128 (matching ColPali for interchangeable indices). 7B model: 2× slower than ColPali but 5–10 NDCG points higher on multilingual benchmarks. Use ColQwen2-VL 2B as drop-in ColPali replacement with better multilingual; use 7B when accuracy dominates cost. [Byaldi](https://github.com/AnswerDotAI/byaldi) (Answer.AI) wraps ColPali/ColQwen with simplified indexing and search. Handles PDF→image, embedding, indexing, and querying. CLI: `byaldi index my_index pdfs/`, `byaldi search my_index "query"`. Built-in support for Hugging Face datasets for large-scale indexing, multi-GPU acceleration, and incremental index updates. Best for teams wanting Visual RAG without building custom indexing infrastructure. [Qdrant](/tools/qdrant) with multi-vector support (v1.9+) is the preferred vector database for Visual RAG. Stores ColPali/ColQwen multi-vectors natively — no manual vector packing/unpacking. Binary quantization reduces storage 4× with <5% retrieval quality loss. Search API: `client.search(collection_name, query_vector=query_multi_vec, limit=10)` returns top-k page IDs. Combine with [pgvector](/tools/pgvector) for metadata filtering: store document metadata (date, author, source) in PostgreSQL, retrieve via metadata filter + vector similarity hybrid search.

Setup walkthrough

  1. pip install colpali-engine (ColPali — retrieve documents by visual appearance, no OCR needed).
  2. Download the model: ColPali/ColQwen2 will auto-download on first use (~2-5 GB).
  3. Index your PDF documents as page images:
from colpali_engine import ColPali
import fitz  # PyMuPDF
model = ColPali.from_pretrained("vidore/colpali-v1.2")
pdf = fitz.open("report.pdf")
page_embeddings = []
for page in pdf:
    pix = page.get_pixmap(dpi=200)
    img = pix.tobytes("png")
    emb = model.embed_images([img])  # multi-vector embedding per page
    page_embeddings.append(emb)
  1. Query: query_emb = model.embed_queries(["Q3 revenue chart"]) → similarity search over page embeddings → returns the most visually-similar pages.
  2. First retrieval in <1 second per query on GPU, ~5 seconds on CPU for a 1,000-page corpus.
  3. The key advantage: ColPali retrieves based on visual layout, not OCR text. It finds the page with the revenue chart even if "revenue" isn't in the OCR text (embedded in the chart image).

The cheap setup

Visual RAG is surprisingly hardware-friendly. ColPali (Vision Transformer based) indexes 5-10 pages/second on CPU, ~50-100 pages/second on a used GTX 1060 6 GB ($60). A 10,000-page corpus indexes in ~15-30 minutes on a $100 GPU. Retrieval is sub-second on any hardware. Pair with Ryzen 5 5600 + 32 GB DDR4 + 1TB NVMe (for storing page images). Total: ~$320-370. For the full pipeline (retrieve pages → VLM answers), add 12 GB VRAM for a 7B VLM. Total with VLM: ~$400-500. Visual RAG is one of the most practical $300-500 local AI setups.

The serious setup

Used RTX 3090 24 GB (~$700-900, see /hardware/rtx-3090). Runs the full Visual RAG pipeline end-to-end: ColQwen2 indexing at 200+ pages/second, retrieval in <100ms, Qwen2-VL 72B for answer generation. Handles 1M+ page corpora with sub-second query latency. For enterprise Visual RAG (legal document archives, scientific paper libraries): combine with a vector DB (Qdrant) that supports multi-vector ColBERT-style indexing. Total: ~$1,800-2,200. Visual RAG eliminates the brittle OCR step entirely — a paradigm shift for document retrieval.

Common beginner mistake

The mistake: Running OCR on every page, embedding the OCR text, and calling it "visual RAG" — then being confused why the system can't find a page with an embedded chart. Why it fails: OCR extracts text, not visual structure. A chart labeled "Q3 Revenue" with bars showing $10M, $12M, $15M might OCR as "Q3 Revenue $10M $12M $15M" with no indication these are values on a chart. The embedding of that OCR text is similar to any page mentioning "Q3 Revenue" — it can't distinguish a chart from a text mention. The fix: Use true Visual RAG (ColPali/ColQwen) that embeds the page image directly. The multi-vector embedding captures visual patterns — a chart page looks visually different from a text page, even if the OCR text is similar. ColPali finds the chart because it "sees" bars and axes, not because it reads "revenue."

Recommended setup for visual rag

Recommended runtimes

Browse all tools for runtimes that fit this workload.

Budget build
AI PC under $1,000 →
Best GPU for this task
Best GPU for local AI →

Reality check

Local AI workloads have real hardware constraints that vary by task type. VRAM ceiling decides what model fits; bandwidth decides decode speed; compute decides prefill speed. Pick the GPU tier that fits your actual workload, not the spec sheet.

Common mistakes

  • Buying for spec-sheet VRAM without modeling KV cache + activation overhead
  • Underestimating quantization quality loss below Q4
  • Skipping flash-attention support (real perf gap on long context)
  • Ignoring sustained-load thermals (laptops thermal-throttle within 30 min)

What breaks first

The errors most operators hit when running visual rag locally. Each links to a diagnose+fix walkthrough.

Before you buy

Verify your specific hardware can handle visual rag before committing money.

Hardware buying guidance for Visual RAG

RAG workflows mix embedding throughput, long-context inference, and reasonable VRAM headroom. The guides below cover the buyer decision honestly.

Related tasks

Document Understanding
Buyer guides
Compare hardware
Troubleshooting
Specialized buyer guides
Updated 2026 roundup