# LLM Evaluation Harness: Production-Grade Testing I shipped a "better" version of my RAG system last month. Users complained it was worse. Turns out I had no way to measure quality—I was just guessing. Traditional ML metrics don't work for LLMs. BLEU scores are useless. ROUGE is slightly less useless but still bad. Here's how I built an evaluation system that actually works. For comprehensive evaluation frameworks, see [Judging LLM-as-a-Judge with MT-Bench](https://arxiv.org/abs/2306.05685) [1] and [HELM: Holistic Evaluation of Language Models](https://arxiv.org/abs/2211.09110) [2]. For practical patterns, see [LangChain Evaluation Documentation](https://python.langchain.com/docs/guides/evaluation/) [3] and [RAG Evaluation Best Practices](https://www.pinecone.io/learn/rag-evaluation/) [4]. ## Evaluation Architecture B[Evaluation Engine] B --> C[Offline Evaluation Pipeline] B --> D[Online Evaluation Pipeline] C --> E[Gold Set Testing] E --> F[Exact Match/F1/BLEU/ROUGE] E --> G[Semantic Similarity] C --> H[LLM-as-Judge - GPT-4/Claude] H --> I[Score + Reasoning] C --> J[Regression Test Suite] J --> K[Baseline Comparison] K --> L{Performance Drop?} L -->|Yes| M[Block Deployment] L -->|No| N[Allow Deployment] D --> O[A/B Testing] O --> P[Variant Assignment] P --> Q[Metrics Collection] D --> R[Human Feedback Loop] R --> S[Rating Collection 1-5] S --> T{Low Rating?} T -->|Yes| U[Alert + Review Queue] T -->|No| V[Log Success] D --> W[Production Metrics] W --> X[Retrieval Metrics - MRR/NDCG] W --> Y[Generation Metrics] W --> Z[System Metrics - Latency/Cost] F --> AA[Results Aggregation] G --> AA I --> AA Q --> AA S --> AA X --> AA Y --> AA Z --> AA AA --> AB[Metrics Dashboard] AA --> AC[Alerting System] AA --> AD[CI/CD Integration] style A fill:#8B5CF6 style AA fill:#00F5FF style AB fill:#8B5CF6 style M fill:#ff4444 style U fill:#ff4444 `} /> ### Official Evaluation Frameworks ## Evaluation Types ### 1. Offline Evaluation Run on fixed datasets before deployment. #### Gold Set Testing **What it is**: Pre-annotated test cases with expected outputs. **Structure**: ```python gold_set = [ { "id": "test_001", "input": "What is RAG?", "expected_output": "Retrieval-Augmented Generation combines...", "context": ["doc1", "doc2"], "metadata": { "category": "definition", "difficulty": "easy" } } ] ``` **Metrics**: - **Exact Match**: Output matches exactly - **F1 Score**: Token-level overlap - **BLEU**: Translation quality metric - **ROUGE**: Summarization metric - **Semantic Similarity**: Embedding-based ```python from sentence_transformers import SentenceTransformer import numpy as np model = SentenceTransformer('all-MiniLM-L6-v2') def semantic_similarity(predicted: str, expected: str) -> float: pred_emb = model.encode(predicted) exp_emb = model.encode(expected) return np.dot(pred_emb, exp_emb) / ( np.linalg.norm(pred_emb) * np.linalg.norm(exp_emb) ) ``` ### 2. LLM-as-Judge Use a more powerful LLM to evaluate outputs. B[System Under Test] B --> C[Generated Output] C --> D[Judge LLM] A --> D E[Expected Output] --> D D --> F[Score + Reasoning] style A fill:#8B5CF6 style C fill:#8B5CF6 style F fill:#00F5FF `} /> **Judge Prompt**: ```python JUDGE_PROMPT = """You are an expert evaluator. Rate the following response on a scale of 1-10. Question: {question} Expected Answer: {expected} Generated Answer: {generated} Evaluate based on: 1. Accuracy (0-3 points) 2. Completeness (0-3 points) 3. Clarity (0-2 points) 4. Relevance (0-2 points) Provide a score and brief reasoning.""" def llm_as_judge(question: str, expected: str, generated: str) -> Dict: prompt = JUDGE_PROMPT.format( question=question, expected=expected, generated=generated ) response = openai.ChatCompletion.create( model="gpt-4", messages=[{"role": "user", "content": prompt}], temperature=0 ) return parse_judge_response(response.choices[0].message.content) ``` **Best Practices**: - Use GPT-4 or Claude as judge (more reliable) - Provide clear evaluation criteria - Use few-shot examples - Ask for reasoning, not just scores ### 3. Regression Testing Catch regressions before they hit production. ```python class RegressionTest: def __init__(self): self.baseline_results = {} self.current_results = {} def run_test_suite(self, test_cases: List[Dict]) -> Dict: results = {} for test in test_cases: output = self.system.generate(test["input"]) score = self.evaluate(output, test["expected"]) results[test["id"]] = score return results def check_regression(self, threshold: float = 0.05) -> bool: """Check if performance dropped by more than threshold""" baseline_avg = np.mean(list(self.baseline_results.values())) current_avg = np.mean(list(self.current_results.values())) drop = (baseline_avg - current_avg) / baseline_avg return drop > threshold ``` ### 4. Online Evaluation Monitor performance in production. #### A/B Testing ```python def ab_test(user_id: str, variants: List[str]) -> str: """Assign user to variant based on consistent hashing""" hash_value = hash(f"{user_id}_{experiment_id}") % 100 cumulative = 0 for variant, percentage in variants: cumulative += percentage if hash_value < cumulative: return variant return variants[-1][0] # Default ``` #### Human Feedback Loop B[LLM Response] B --> C[User Feedback] C --> D{Positive?} D -->|Yes| E[Log Success] D -->|No| F[Log Issue] F --> G[Review Queue] G --> H[Human Review] H --> I[Update Gold Set] style A fill:#8B5CF6 style E fill:#00F5FF style F fill:#ff4444 `} /> **Feedback Collection**: ```python def collect_feedback( query: str, response: str, rating: int, # 1-5 feedback_text: str = None ): feedback = { "query": query, "response": response, "rating": rating, "feedback": feedback_text, "timestamp": datetime.now(), "model_version": get_model_version() } # Store for analysis feedback_db.insert(feedback) # Alert on low ratings if rating <= 2: alert_manager.send_alert({ "type": "low_rating", "query": query, "response": response }) ``` ## RAG-Specific Evaluation ### Retrieval Metrics **Precision@K**: Fraction of retrieved docs that are relevant ```python def precision_at_k(retrieved: List[str], relevant: List[str], k: int) -> float: top_k = retrieved[:k] relevant_retrieved = len(set(top_k) & set(relevant)) return relevant_retrieved / k if k > 0 else 0 ``` **Recall@K**: Fraction of relevant docs that were retrieved ```python def recall_at_k(retrieved: List[str], relevant: List[str], k: int) -> float: top_k = retrieved[:k] relevant_retrieved = len(set(top_k) & set(relevant)) return relevant_retrieved / len(relevant) if relevant else 0 ``` **MRR (Mean Reciprocal Rank)**: Average of 1/rank of first relevant result ```python def mrr(queries: List[Dict]) -> float: reciprocal_ranks = [] for query in queries: retrieved = query["retrieved"] relevant = set(query["relevant"]) for rank, doc_id in enumerate(retrieved, 1): if doc_id in relevant: reciprocal_ranks.append(1.0 / rank) break else: reciprocal_ranks.append(0.0) return np.mean(reciprocal_ranks) ``` ### Generation Metrics **Answer Correctness**: Does the answer address the question? **Context Relevance**: Is the answer grounded in retrieved context? **Citation Accuracy**: Are citations correct? ```python def evaluate_rag_response( question: str, answer: str, retrieved_context: List[str], citations: List[Dict] ) -> Dict: scores = { "answer_correctness": llm_as_judge( question=question, expected="[Answer should address the question]", generated=answer ), "context_relevance": check_context_grounding(answer, retrieved_context), "citation_accuracy": verify_citations(answer, citations, retrieved_context) } return scores ``` ## Evaluation Framework ### Test Suite Structure ```python class EvaluationHarness: def __init__(self): self.gold_set = load_gold_set() self.metrics = { "semantic_similarity": semantic_similarity, "llm_judge": llm_as_judge, "retrieval_metrics": compute_retrieval_metrics } def run_evaluation(self, system) -> Dict: results = { "overall": {}, "by_category": {}, "regression": {} } for test_case in self.gold_set: output = system.generate(test_case["input"]) scores = self.compute_all_metrics( output, test_case["expected"], test_case ) results["overall"][test_case["id"]] = scores category = test_case["metadata"]["category"] if category not in results["by_category"]: results["by_category"][category] = [] results["by_category"][category].append(scores) # Aggregate results["overall"]["average"] = self.aggregate_scores( list(results["overall"].values()) ) # Check regression results["regression"] = self.check_regression(results) return results ``` ### Continuous Evaluation B[CI/CD Pipeline] B --> C[Run Test Suite] C --> D{Pass?} D -->|No| E[Block Deployment] D -->|Yes| F[Deploy] F --> G[Production] G --> H[Online Evaluation] H --> I[Metrics Dashboard] I --> J{Regression?} J -->|Yes| K[Alert + Rollback] J -->|No| L[Continue] style A fill:#8B5CF6 style F fill:#00F5FF style E fill:#ff4444 style K fill:#ff4444 `} /> ## Production Monitoring ### Key Metrics 1. **Quality Metrics**: - Average LLM-as-judge score - Human feedback rating - Error rate 2. **Retrieval Metrics**: - Precision@K, Recall@K - MRR, NDCG - Average retrieval latency 3. **Generation Metrics**: - Answer correctness - Citation accuracy - Hallucination rate 4. **System Metrics**: - Latency (p50, p95, p99) - Throughput - Error rate - Cost per query ### Alerting ```python def check_evaluation_alerts(metrics: Dict): # Quality drop if metrics["avg_score"] < baseline_score * 0.95: alert("Quality degradation detected") # Regression if metrics["regression_detected"]: alert("Regression in test suite") # High error rate if metrics["error_rate"] > 0.05: alert("High error rate in production") ``` ## Best Practices 1. **Start Simple**: Begin with basic metrics, add complexity as needed 2. **Automate**: Run evaluations in CI/CD 3. **Monitor Continuously**: Track metrics in production 4. **Human-in-the-Loop**: Combine automated and human evaluation 5. **Version Control**: Track model versions and evaluation results 6. **Baseline Comparison**: Always compare against a baseline 7. **Category Breakdown**: Evaluate by category, not just overall ## Implementation Checklist - [ ] Gold set creation and maintenance - [ ] LLM-as-judge implementation - [ ] Regression test suite - [ ] Online evaluation pipeline - [ ] Human feedback collection - [ ] Metrics dashboard - [ ] Alerting system - [ ] CI/CD integration - [ ] Baseline tracking - [ ] Documentation ## Remember Evaluation isn't a one-time thing. It's continuous. Build it into your workflow from day one, and your system will actually get better over time.