Our risk committee asked a straightforward question about an AI feature with no LLM Evaluation Framework behind it we wanted to release. What evidence do you have that this works, and would it stand up in six months if somebody challenged a decision it influenced?
We did not have an answer. Building one took a quarter and it is now how every AI feature reaches production.
Assurance is not the same as testing
Testing asks whether it works now. Assurance asks whether you can demonstrate that, later, to somebody sceptical.
The difference shows up in what you keep. A test run tells you today's result. Assurance requires the test set, the criteria, the scores, the versions of everything involved, and a record of who approved it.
Our first evaluation had none of that. The numbers were real and entirely unverifiable after the fact.
Pin and record every version
A result means nothing without knowing what produced it.
We record the model version, the prompt, the retrieval configuration, the document set, and the code, each with an identifier. Any historic result can be reproduced.
That mattered the first time a model provider updated a version under us and our scores moved. Without pinning, we would have been investigating our own code for a change we had not made.
Keep the criteria stable and dated
Criteria change, and when they do, history breaks.
Ours are versioned. When a criterion changes, that is recorded with a date and a reason, and results before and after are not compared directly.
An LLM Evaluation Framework that quietly adjusts its standards will always appear to be improving.
Measure the things a reviewer will ask about
Accuracy is the obvious one and rarely the one that causes trouble.
We also measure correct refusal, whether sources are cited and whether those citations are real, consistency across repeated runs of the same input, and whether outcomes differ across customer groups in ways that cannot be justified.
That last measure has twice prompted an investigation. Both turned out to be artefacts of the test set composition, and we could show the working.
Record what the human reviewer did
Human oversight is only evidence if it leaves a trace.
For every output reaching a customer, we log who reviewed it, when, whether they changed it, and what they changed. That record is what turns a claim of human review into something demonstrable.
It also produces a useful measure. A rising edit rate means the system is drifting, and a falling one towards zero may mean reviewers have stopped reading.
Separate the model risk from the process risk
A distinction our risk committee asked for and we now find useful.
Model risk is the system being wrong. Process risk is nobody noticing, or noticing and being unable to act.
Most of what we built addresses the second. The model will sometimes be wrong, which is accepted and quantified. What is not acceptable is being unable to tell, unable to reproduce, or unable to stop it.
Framing the pack that way changed the conversation with the committee entirely, because the question stopped being whether the model is perfect.
Decide the failure response in advance
Assurance includes knowing what happens when it goes wrong.
Ours names who can disable the feature, how quickly, what the fallback process is, how affected cases are identified, and who tells the customer.
We rehearse it annually. The first rehearsal found that identifying affected cases would have taken two days, because we were not logging enough to find them.
Keep the pack readable by non engineers
The audience is the point.
Our first attempt was a set of dashboards and a repository. The risk committee could not use any of it, which meant the evidence existed and did not function as evidence.
The pack is now eight pages. What the system does, what good means, how it scored, what it refuses, who reviews it, and what happens when it fails.
An LLM Evaluation Framework whose output only engineers can interpret has not finished the job it was built for.
Watch production, not just the test set
A fixed test set tells you about a fixed world.
We sample live cases monthly, score them against the same criteria, and compare with the test set results. Divergence means the test set has aged.
Ours needed refreshing after eight months because the documents customers sent had changed shape. Nothing about the system had degraded, and it was performing worse on real work.
Cover what it connects to
A grounded system inherits the quality of its sources.
Ours reads records from salesforce cloud solutions, and our early failures were mostly retrieval problems. Permissions not respected, stale records returned, and the system answering confidently from an incomplete picture.
Those cases are now in the test set explicitly, which means a permissions regression shows up as an evaluation failure rather than as a customer complaint. Automation Testing Services cover the same boundary for ordinary integrations, and the reasoning is identical.
What our assurance pack contains
- Versioned, dated criteria per task.
- A fixed real case test set, with refresh history.
- Pinned versions for model, prompt, retrieval, data and code.
- Scores including refusal, citation validity and consistency.
- Outcome comparison across customer groups.
- Human review logs with edit rates.
- A rehearsed failure response with named owners.
Common questions
Is this needed for every feature?
Scaled to impact. Anything affecting a customer decision gets the full pack. Internal drafting tools get less.
How long does it take to build?
Our first took a quarter. Each one since has taken about three weeks, because the framework exists.
Can the evaluation be automated?
The running of it, yes. The criteria and the calibration need people. Automation Testing Services face the same boundary, where a machine runs a check and a person decides what good means.
Who should own the framework?
A named owner outside the delivery team. An LLM Evaluation Framework owned by the people shipping the feature will drift towards measuring what passes.
What do reviewers ask about most?
Reproducibility. An LLM Evaluation Framework that cannot reconstruct a six month old result will not satisfy anybody.
Where this leaves us
Every AI feature now arrives at the risk committee with the same pack, and the conversation takes twenty minutes rather than three meetings.
The pack is not sophisticated. Written criteria, real cases, pinned versions, logged reviews, and a rehearsed plan for failure. What makes it work is that all of it exists before release rather than being assembled afterwards from whatever happens to have been kept.
