Advanced RAG — Chunking, Retrieval, Re-ranking

22. RAGAS Answer Relevance

Chapter 22 of 24 · 20 min

KEY INSIGHT

Answer relevance measures how directly the generated answer addresses the specific question asked, penalizing tangents and tangential verbosity. ### Definition An answer has relevance when its content directly addresses all aspects of the question. A question asking "compare latency of Kafka and RabbitMQ" that produces an answer about throughput only scores low on relevance even if factually correct. RAGAS computes relevance by generating question-like probes from the answer and measuring cosine similarity between probe questions and the original question. ### Implementation ```python import numpy as np def generateQuestionProbes( answer: str, numProbes: int = 3, model: str = "gpt-4o-mini" ) -> list[str]: """ Generate alternative questions that the answer would answer. """ prompt = ( f"Given the answer below, generate {numProbes} different questions " "that this answer would respond to. The questions should vary in phrasing " "but all be answerable by the provided answer content. " "Return one question per line.\n\nAnswer:\n{answer}" ) response = client.chat.completions.create( model=model, messages=[ {"role": "system", "content": "Be concise and varied in phrasing."}, {"role": "user", "content": prompt} ], temperature=0.8, # Some diversity in probe generation max_tokens=256 ) probes = [p.strip() for p in response.choices[0].message.content.split("\n") if p.strip()] return probes[:numProbes] def computeAnswerRelevance( originalQuestion: str, answer: str, embeddingModel ) -> dict: probes = generateQuestionProbes(answer, numProbes=3) # Compute similarity between original question and each probe originalEmb = embeddingModel.encode(originalQuestion) similarities = [] for probe in probes: probeEmb = embeddingModel.encode(probe) sim = cosineSimilarity(originalEmb, probeEmb) similarities.append(sim) relevanceScore = np.mean(similarities) return { "relevance_score": round(relevanceScore, 3), "probes": probes, "probe_similarities": [round(s, 3) for s in similarities] } ``` ### Handling Long Answers Verbose answers that include correct information plus tangents score lower, because probe generation from verbose answers produces probes that don't match the original question. This penalizes the right behavior: ```python def computeRelevanceOnDataset( dataset: list[dict], embeddingModel ) -> dict: scores = [] for item in dataset: result = computeAnswerRelevance( item["question"], item["answer"], embeddingModel ) scores.append(result["relevance_score"]) return { "mean_relevance": mean(scores), "scores": scores } ``` ### Tuning the Probe Count More probes increase measurement stability but increase API calls and cost. At numProbes=3, standard deviation across repeated runs on the same answer is typically 0.02–0.05, acceptable for production evaluation. At numProbes=1, variance is too high; at numProbes=10, marginal accuracy gain does not justify cost. ### Failure Modes Probe generation can produce questions that the original answer actually does answer correctly (high precision) but that are phrased similarly to each other, reducing diversity and underestimating relevance variance. Use a diversity-promoting instruction in the probe generation prompt. If the answer is completely off-topic, probes describe the off-topic content and similarity to the original question is near zero—no false negatives, but verify with a manual check for face validity.

EXERCISE

Implement answer relevance evaluation. Run on 20 Q&A pairs and correlate relevance scores with manual human ratings. Report Pearson correlation. (15 min)