LLM evaluation — sometimes called "evals" — is the practice of systematically measuring whether your AI application produces outputs that are accurate, safe, consistent, and actually useful for real users. It's the testing infrastructure for AI, and in 2026 it's moved from a research concern to a core engineering discipline. The reason it matters right now: model providers update their models frequently without guarantees of backward compatibility, and a prompt that worked well last quarter may produce measurably worse results after a model update. Without evals, you won't know until users complain.
Why Unit Tests Aren't Enough
You can write unit tests for your AI application's non-LLM code — input parsing, output formatting, routing logic, tool calls. But you can't write a traditional assertion like assert response == "correct answer" for generative outputs. A question about your refund policy might have ten different correct phrasings. A summarization task might have many acceptable outputs and no single ground truth.
This is what makes LLM evaluation distinct from conventional software testing. You're not checking for an exact value — you're assessing quality across multiple dimensions: factual accuracy, completeness, relevance, tone, safety, and often task-specific criteria. Each of those dimensions needs its own measurement approach.
The Three Layers of an Eval Stack
A mature LLM evaluation setup has three layers that serve different purposes and run on different cadences.
Layer 1: Functional correctness on a golden dataset
This is your regression suite. A golden dataset is a collection of representative input-output pairs where the correct output is known and agreed upon. Every time you change a prompt, switch a model, modify your RAG retrieval, or push any change that touches LLM behavior, you run this suite and check for regressions.
Building a useful golden dataset:
- Pull real user queries from your logs or expected use cases — not synthetic examples you invented. Synthetic queries miss the weird, ambiguous phrasing real users actually send.
- Write reference answers for each query, being specific about what "correct" means. Vague reference answers make it impossible to score reliably.
- Include failure cases: queries that should be refused, queries at the edge of your knowledge base, and adversarial inputs that tempt the model to hallucinate.
- Aim for 100–500 examples. Smaller sets miss important cases; larger sets slow your iteration speed without proportional benefit at early stages.
Maintain this dataset in version control alongside your code. When you discover a new failure mode in production, add an example to the golden dataset so you can detect regressions against it in the future.
Layer 2: LLM-as-judge evaluation
For tasks where the output space is too broad for exact-match scoring, use a separate LLM to evaluate your application's outputs. The basic pattern: pass the question, the application's answer, and optionally a reference answer to a judge model (often a more capable or different model than the one generating responses), and ask it to score on specific criteria.
A practical LLM-as-judge prompt structure:
- Criterion-specific rubrics rather than a single quality score. Score factual accuracy separately from completeness separately from tone — you want to know which dimension is degrading, not just that something got worse.
- Numerical scores with defined anchors. "1 = factually incorrect, 2 = partially correct, 3 = fully accurate" is more consistent than "rate quality 1–5."
- Chain-of-thought reasoning before the score. Asking the judge to explain its reasoning before scoring significantly improves score reliability.
Calibrate your judge against human ratings. Take 100 examples, have humans score them, and measure agreement between your judge and the humans. This calibration step catches judge biases (models tend to prefer longer answers, their own outputs, or responses that match their training distribution) before you rely on the scores for decisions.
Layer 3: Production monitoring
Evals that only run in CI catch regressions but miss the full distribution of real user behavior. Production monitoring completes the picture:
- Log every LLM input, output, and metadata (latency, token count, retrieved document IDs, confidence scores).
- Instrument user feedback signals — thumbs up/down, explicit corrections, session abandonment after a specific response.
- Run async LLM-as-judge scoring on a sample of production outputs nightly. Flag outputs below your quality threshold for human review.
- Track drift metrics: if your average quality score drops 10% over a two-week window without any code change, that's a signal worth investigating (possible model update, distribution shift in user queries, knowledge base staleness).
Eval Datasets: How to Build Them Without Spending Months
The most common objection to evals is the time cost of building datasets. Here's a practical approach that gets you to a working eval suite in days, not months:
- Mine existing data first. If your application has any usage history, real queries are already available. If you're pre-launch, generate queries by having team members use the application for a week and logging everything.
- Use your application to generate candidate answers. For each query, run your current application and generate a response. This is your baseline — it's what you're trying to detect regressions against.
- Human review to create ground truth. Have domain experts review the generated responses and label them: correct as-is, correct with edits (provide the edited version), or incorrect (provide the correct answer). This is the time-intensive step but 100 well-labelled examples is usually enough to start.
- Automate incrementally. As your eval suite grows, add automation to catch obvious failure modes (refused answers where an answer was expected, answers that are too short, responses that don't cite sources when required) before running the more expensive LLM-as-judge scoring.
Defining "Good Enough to Ship": Acceptance Criteria for AI Features
One of the harder questions in AI engineering is when to ship. Unlike traditional software where functionality is binary — the button works or it doesn't — AI output quality is a distribution. You need explicit acceptance criteria defined before you start building, not after.
| Eval dimension | Example acceptance threshold | Notes |
|---|---|---|
| Factual accuracy (golden dataset) | ≥ 92% correct on in-scope queries | Calibrate against your domain's risk profile |
| Appropriate refusal | ≥ 95% refusal rate on out-of-scope queries | False negatives (hallucinating instead of refusing) are high-risk |
| Latency | p95 response time ≤ 4 seconds | Time-to-first-token matters for streaming UX |
| Safety / policy compliance | 0 policy violations on adversarial test set | Zero tolerance; any violation blocks ship |
| LLM-judge quality score | Average ≥ 4.0 / 5.0 | Calibrate judge scores against human ratings first |
Document these criteria and get stakeholder sign-off before building. This prevents the common situation where "good enough" keeps shifting as launch pressure mounts.
Regression Testing When Model Providers Update
Model providers update their models on rolling schedules, and those updates can degrade performance on specific task types even when they improve aggregate benchmarks. Pin your model version in production code (e.g., gpt-4o-2024-08-06 rather than gpt-4o) and treat model upgrades like dependency upgrades — run your full golden dataset eval before switching. Keep a version history of golden dataset scores so you can see whether an update helped or hurt your specific use case.
Tools Worth Knowing
Several tools reduce the overhead of building eval infrastructure: Promptfoo for prompt testing and comparison, RAGAS for RAG-specific metrics (faithfulness, answer relevancy, context precision), LangSmith for production tracing and LLM-as-judge at scale. None replace the need for domain-specific golden datasets, but they cut the plumbing work significantly. Engineering teams at Mexilet Technologies who build AI applications for clients typically set up a lightweight eval pipeline in the first sprint — before any user-facing features ship. The investment pays back within the first model provider update cycle, usually within 60–90 days.
Frequently Asked Questions
How many examples do I need in my golden dataset to have meaningful evals?
For most applications, 100–200 well-curated examples gives you a useful signal. The quality of examples matters more than quantity — 100 representative, carefully labelled examples outperform 1,000 synthetic or poorly labelled ones. Grow the dataset over time by adding examples from real production failures and edge cases you encounter. Prioritize coverage of your highest-risk query categories over raw size.
How often should I run evals?
At minimum: on every change that touches prompt text, model version, retrieval configuration, or knowledge base content. In practice, most teams run the golden dataset suite in CI (triggered on PR), run LLM-as-judge scoring nightly on a sample of production outputs, and do a manual review of flagged outputs weekly. The cadence should match how fast you're shipping changes — faster iteration cycles need more frequent eval runs.
Is LLM-as-judge reliable enough to trust for production decisions?
It's reliable enough to detect regressions and surface outputs worth human review — not reliable enough to be the sole arbiter of quality for high-stakes decisions. Calibrate your judge against human ratings before relying on it. Be aware of known biases: LLMs tend to prefer longer responses and responses similar to their own style. Use multiple judges or average across judge runs for higher-stakes evaluations. LLM-as-judge is a scalable filter, not a replacement for human judgment on difficult cases.
What's the difference between evals and monitoring?
Evals are structured, offline tests against a defined dataset — run before shipping a change. Monitoring is continuous, online measurement of production behavior on real traffic. Both are necessary: evals catch regressions before they reach users; monitoring catches problems that evals didn't anticipate, including distribution shifts in real queries and degradation from knowledge base staleness or silent model updates.
Need a partner for this? Mexilet offers generative AI development and AI solutions.
Building reliable AI applications is a team sport — it requires prompt engineers, backend developers, domain experts who can review outputs, and infrastructure for evaluation at scale. If you're looking for an offshore development partner who can help build both the AI features and the eval infrastructure to support them in production, let's start a conversation about how Mexilet can support your team.