Our first Healthcare Generative AI deployment was described as transformative by everybody involved. Nobody could tell me whether it had saved any time.
Measuring our Healthcare Generative AI work took a quarter to set up and changed two decisions immediately. Here is the framework, offered as an account of Healthcare Generative AI governance rather than clinical guidance.
Enthusiasm is not evidence
The uncomfortable starting point.
Staff described the drafting tool as a significant improvement. Usage was high. Feedback was positive. None of that told us whether the clinical day was shorter or whether the record was better.
When we measured, the saving was real and about half what people believed, which is still a worthwhile result and a very different business case.
Measure the current process first
Four weeks, before anything is built.
How long the task takes today, measured rather than estimated. How often it happens. What the current error rate is. And how long the existing review takes.
Two proposals failed at this stage. Both saved less time than the review they would have created, which nobody had calculated because nobody had measured the baseline.
Track the reviewer, not just the output
The measure that tells you most.
Every draft records who reviewed it, how long they spent, and what they changed. The edit rate is the single most informative number we collect.
Rising means the system is drifting or the work has changed. Falling towards zero may mean reviewers have stopped reading carefully, which is a finding rather than a success.
We sample conversations and drafts monthly rather than trusting the trend.
Count correct refusal as success
A criterion frequently omitted.
When the information is not in the record, the right output is to say so. A system that invents a plausible detail is worse than one that declines.
Our test set includes cases where no valid answer exists, and the rate of correct refusal is reported alongside everything else. A candidate system that answered every one of those confidently was ruled out during evaluation.
Measure what it does to the record, not just to time
The question clinical governance asks.
Are discharge summaries more complete? Are they more consistent between clinicians? Do they contain anything that was not in the source record?
That last one must be zero and is checked by sampling. Time saved is the business case. Record quality is the clinical one, and both are reported.
Watch for differential effects
A requirement that transfers from any clinical system.
We compare outputs across patient groups and investigate differences we cannot account for. Twice this has prompted a review, and both were artefacts of the sample rather than the system.
Being able to show that working matters as much as the result.
Measure what the data allows
A ceiling set before any model is chosen.
Our records had been through a migration that documented field definitions and completeness. Where that documentation was thin, output quality was measurably worse, and no model change improved it.
Data Migration Services work of that kind sets the upper bound on what any later deployment can achieve, which is worth knowing before a business case is written.
The data underneath decides the ceiling
Where measurement usually leads.
When output quality is poor, the cause is nearly always what the system could see rather than how it reasoned. Incomplete records, a document not indexed, permissions preventing retrieval.
Our estate has been through an architectural programme applying the usual thinking on moving from monolith to microservices, which made records reachable through stable interfaces. Measurement before that work mostly measured the integration.
Keep the sample size honest
Small numbers produce confident wrong conclusions.
Our first month showed a dramatic improvement on forty cases. The quarter showed something more modest on nine hundred.
We now report the sample size beside every figure and refuse to draw conclusions below a stated threshold. That prevented two decisions the early numbers would have justified.
Report it where decisions are made
Monthly, one page, including what did not work.
Time against baseline, edit rate, correct refusals, record quality sampling, and incidents. Those five go to the clinical governance group and the programme board.
Reporting the disappointing results is what made the positive ones credible. The first report that recommended stopping something was the first one anybody read closely.
What we measure for every deployment
- A baseline measured before the build.
- Time per task against that baseline.
- Reviewer edit rate, in both directions.
- Correct refusal rate on cases with no valid answer.
- Record completeness and consistency, sampled.
- Anything appearing in output that was not in the source, which must be zero.
- Differential outcomes across patient groups.
Common questions
Does scope affect measurability?
Considerably. A narrow Healthcare Generative AI deployment with one task and one reviewer is straightforward to measure; a broad one is not.
How long until a result is meaningful?
A quarter for a usable signal, a year for confidence. Early enthusiasm tells you nothing.
Who should own the measurement?
The organisation, against criteria set beforehand. Healthcare Generative AI suppliers can provide the data and should not define what good looks like.
Should the supplier measure it?
They can provide data. The measurement and the judgement belong to the organisation, against criteria set beforehand.
What if the numbers are disappointing?
Report them. We stopped one deployment on that basis and the framework was trusted afterwards because of it.
What limits the result most?
The data. Our Data Migration Services programme documented definitions and completeness, and output quality tracked that documentation closely.
What is most often overstated?
Time saved. Healthcare Generative AI proposals tend to compare against an idealised current process rather than the measured one.
Where this left us
Four deployments, three clearly worthwhile, one stopped, and a framework that produces the same five numbers every month.
None of that required sophisticated measurement. It required deciding what good looked like before building, measuring the current process honestly, and being willing to report a result nobody wanted.
