16. Performance Optimization

Chapter 16 of 18 · 20 min

KEY INSIGHT

The main bottlenecks are embedding computation and vector searchâ€”optimize embeddings first, then database operations. ### Profiling Before optimizing, measure where time goes: ```python import time from functools import wraps def profile(func): """Decorator to time function execution.""" @wraps(func) def wrapper(*args, **kwargs): start = time.time() result = func(*args, **kwargs) elapsed = time.time() - start print(f"{func.__name__}: {elapsed:.3f}s") return result return wrapper class ProfilingSearchEngine(SemanticSearchEngine): @profile def index_documents(self, documents, ids=None, metadatas=None): return super().index_documents(documents, ids, metadatas) @profile def search(self, query, top_k=5, filters=None): return super().search(query, top_k, filters) ``` ### Optimizing Embedding Speed ```python # 1. Use batch encoding (already default in ChromaDB) batch_embeddings = model.encode(docs, batch_size=256, show_progress_bar=True) # 2. Pre-encode common queries COMMON_QUERIES = [ "How do I reset my password?", "Where can I find my invoice?", "How do I contact support?" ] query_cache = {q: model.encode(q) for q in COMMON_QUERIES} def cached_query(query): if query in query_cache: return query_cache[query] return model.encode(query) # 3. Use half-precision for storage (reduce memory) embeddings_fp16 = embeddings.astype('float16') # Half the memory ``` ### Optimizing ChromaDB Queries ```python # 1. Limit result set to what you need results = collection.query( query_texts=[query], n_results=10, # Don't request more than you need ) # 2. Only include fields you use results = collection.query( query_texts=[query], n_results=5, include=["documents"] # Skip metadata and distances if not needed ) # 3. Use approximate search for large collections collection = client.create_collection( name="large_collection", embedding_function=model, metadata={"hnsw:construction_ef": 100, "hnsw:search_ef": 100} ) ``` ### Optimizing FAISS ```python # 1. Use IVFPQ for compression with large datasets nlist = 100 # Number of clusters m = 16 # Number of subquantizers index = faiss.IndexIVFPQ(quantizer, dimension, nlist, m, 8) # 8 bits per vector # 2. Tune nprobe for IVF indexes # Start with sqrt(nlist), adjust based on recall requirements index.nprobe = 10 # Default index.nprobe = 50 # Higher recall, lower speed # 3. Use index_cpu_to_gpu for GPU acceleration if faiss.get_num_gpus() > 0: gpu_index = faiss.index_cpu_to_gpu(res, 0, index) ``` ### Memory Optimization ```python # Monitor memory usage import psutil import os def get_memory_mb(): process = psutil.Process(os.getpid()) return process.memory_info().rss / 1024 / 1024 print(f"Memory before: {get_memory_mb():.1f} MB") # Clear ChromaDB client to free memory del collection del client print(f"Memory after: {get_memory_mb():.1f} MB") ```

EXERCISE

Profile your search engine's indexing and query operations. Identify the bottleneck. Apply one optimization and measure the improvement.