To evaluate LLM output quality in production, log every response with the context that produced it, run fast automated checks on all of it, and grade a steady sample against a written rubric. Use a model as the grader only after you've compared its verdicts with human labels. Then track the rates per feature each week and turn every confirmed failure into a regression test.
Pre-launch evaluation tells you a large language model (LLM) feature worked on the cases you thought of. Production shows what users actually ask.
Why pre-launch evals don't cover production
A pre-launch suite is a fixed set of inputs. Real traffic moves away from it. Users ask in other languages, paste whole contracts, or use the feature for a job it wasn't built for. The system changes too: someone edits a prompt template, new documents enter the index, or the provider ships a new model version.
None of these raise errors. The feature keeps returning fluent text while its quality shifts.
The two kinds of evaluation feed each other. A pre-launch test suite for an LLM feature covers security and misuse before go-live. Production review supplies new cases for it.
What to log for every LLM response
You can't grade what you didn't record. For each response, store enough to replay it:
- a request ID, the feature name and a pseudonymous user or tenant ID
- the model ID and version, and the prompt template version
- the IDs of retrieved documents, and any tool calls with their arguments
- the final output, the finish reason and the token counts
- what the user did next: copied, edited, regenerated, escalated or left
The finish reason deserves its own alert. A response that hit the token limit can look complete in a log viewer and still end mid-sentence.
Prompt and output logs often contain personal data. Decide retention, redaction and read access before you turn logging on.
Which quality signals to track
No single signal tells you whether answers are good. Combine lightweight signals on all traffic with slower, deeper ones on a sample.
| Signal | Coverage | What it catches | Limitation |
|---|---|---|---|
| Automated checks | Every response | Invalid JSON, broken citations, truncation, wrong language, blocked content | Says nothing about whether the answer is right |
| Explicit feedback | Low | Answers users found unhelpful | Clicks are self-selected; silence means little |
| Implicit behavior | Every response | Regenerations, heavy edits, quick rephrasing, escalation to a human | Needs a definition per feature |
| Model-graded sample | A daily or weekly sample | Unsupported claims, missed questions, policy breaks | Only as good as its calibration |
| Human review | A small sample | Subtle errors and new failure types | Slow, and reviewers drift too |
| Business outcome | Delayed | Reopened tickets, reversed decisions | Arrives late and has other causes |
Implicit signals carry a lot of information for little effort. For a feature that drafts support replies, the edit distance between the draft and the reply the agent actually sent measures usefulness directly.
How to sample production traffic for review
Take two samples and keep them apart.
The first is random: a fixed number of responses per feature each week, drawn without looking at them. It's the only sample that gives you an honest failure rate, so don't top it up with interesting cases.
The second is targeted. Pull responses where something looked off: a thumbs-down, a regeneration, an escalation, a failed automated check or a rare request type. This sample finds new failure modes fast, but its failure rate is useless as a metric, because you picked those responses for looking suspicious.
Split both samples by feature and by segments such as language or customer tier. A product-wide average can stay flat while one segment gets much worse.
How to evaluate LLM output quality in production with a model grader
Write the rubric first. For each feature, list a handful of yes-or-no criteria that a domain expert could apply without debate. A hypothetical rubric for an internal policy assistant:
- Did it answer the question that was asked?
- Is every factual claim supported by the retrieved documents?
- Did it say so when the documents didn't contain the answer?
- Did it stay within the topics it's allowed to cover?
Binary criteria are easier to grade consistently than a ten-point scale. Anthropic's guidance on building evaluations recommends detailed rubrics and specific outputs such as "correct" or "incorrect", and advises testing an LLM grader's reliability before scaling it.
Calibrate before you automate:
- Have a domain expert label a few hundred production responses against the rubric.
- Run the grader on the same responses, with the same retrieved context the generator saw.
- Measure agreement per criterion, not overall.
- Automate only the criteria where agreement is high enough to act on. Leave the rest to human reviewers.
- Pin the grader's model version and recalibrate when it changes.
A grader that sees only the question and the answer can't judge groundedness, so give it the sources. Give it its own prompt and, where possible, a different model, so it doesn't share the generator's blind spots.
Measure retrieval separately from the answer
For features built on retrieval-augmented generation (RAG), a bad answer has two possible causes: the right document wasn't retrieved, or it was and the model misused it. The logged document IDs tell a reviewer which. Track the two rates separately. They have different fixes, as explained in choosing between RAG and fine-tuning, and no prompt change will repair a retrieval problem.
Turn production failures into regression tests
Every confirmed failure should become a test case: the input, the context the model saw and a pass rule. Strip personal data first. Run the set before any change to the model, prompt template, retrieval settings or tools, and change one of those at a time so a shift in results has one explanation.
After a few months this set reflects what your users actually ask, which no pre-launch suite can. Model upgrades are the obvious moment to run it. The same pattern explains why AI automations break after a provider update: nothing errors, and the output changes underneath you.
How to detect LLM quality drift
Plot each metric weekly, per feature, and mark every deploy, prompt change and model change on the same chart. When a line moves, the annotation often explains it.
The National Institute of Standards and Technology (NIST) AI Risk Management Framework Playbook covers this under MEASURE 2.4, which suggests documenting how production metrics differ from pre-deployment testing and using control limits to spot anomalies. In practice, set a band around each rate from its first stable weeks and alert when the rate leaves it. Good alert candidates:
- schema or format failure rate
- truncation rate
- refusal rate on requests the feature should handle
- grader failure rate on groundedness
- regeneration or escalation rate
Each alert needs an owner who reads the underlying responses.
Common mistakes when measuring LLM quality in production
- Treating the thumbs-up rate as a quality score. The users who click aren't a random sample.
- Averaging across features. A drafting assistant and a classifier fail in different ways and need separate rubrics.
- Trusting an uncalibrated grader. Its scores can look stable while it disagrees with your experts.
- Changing the model and the prompt in the same release, so nobody can say which one moved the numbers.
- Logging everything and reviewing nothing. Stored prompts without a weekly review add privacy exposure and give you no quality signal.
If you want senior help putting this in place, our LLM app, agent and RAG development work includes evaluation suites, quality metrics and monitoring dashboards for production systems.
Frequently asked questions
How many production responses should we review each week?
Enough that the failure types you care about actually appear. For a low-volume internal tool, that can be every response. For a high-volume feature, start with a few dozen random responses per feature plus the targeted sample.
Can an LLM grader replace human review?
For criteria where it agrees with your experts, it can handle the volume. A person should still review a small sample each week to catch new failure types and grader drift. Keep high-stakes outputs, such as legal, medical or financial answers, with human reviewers.
Can we log user prompts for evaluation under GDPR?
It depends on your lawful basis, what the prompts contain and what your privacy notice tells users. The General Data Protection Regulation (GDPR) also expects data minimization and limited retention, so pseudonymize identifiers and redact where you can. This is general information, not legal advice.
What's the difference between offline and online LLM evaluation?
Offline evaluation runs a fixed test set against a version of the system before release. Online evaluation measures live traffic after release. Offline catches regressions before users see them, and online finds the cases your test set doesn't contain yet.