By Anant Agrawal, Head of AI, CIMCON Software. Based on my talk at ISPE on reviewing GenAI outputs
The demo took ten minutes. The review took two hours.
That gap is the most important number in your GenAI business case, and it is the one most vendors leave out.
The claim you will hear
Ask almost any GenAI vendor about return on investment and you will hear a version of the same answer: our tool drafts the document in minutes instead of hours. A task that took four hours by hand now takes ten minutes, which is a saving of 96%.
That figure measures one thing, the time to generate. In a regulated setting, generating the output is not the end of the job. Someone qualified has to read it, check it against the source, correct it and approve it before it goes anywhere that matters. That time is real, it is spent by your most experienced people, and it rarely appears in the pitch.
The missing line item: time to review
In pharma and GxP, a person remains accountable for what the AI produces. A draft that reads well but contains a wrong value, a missing requirement or a statement the source does not support is worse than no draft, because it looks finished.
So the honest measure of return is the total time to an approved document: time to generate plus time to review, plus any rework. Here is what that looks like in an illustrative example.
The math
These numbers are illustrative, not measured results. Assume a team produces 100 documents a month, and each takes 240 minutes to write by hand.
Table 1: The vendor claim against a realistic review
| Manual | Vendor claim | With realistic review | |
|---|---|---|---|
| Time to generate | 240 min | 10 min | 10 min |
| Time to review | Included | Not counted | 120 min |
| Total per document | 240 min | 10 min | 130 min |
| Monthly hours (100 documents) | 400 | 17 | 217 |
| Saving against manual | n/a | 96% | 46% |
The saving is still real, but it is less than half of what the headline suggested. Review time is the swing factor, so here is how sensitive the result is to it.
Table 2: How the saving moves with review time (generation fixed at 10 minutes)
| Review time per document | Total per document | Saving against manual |
|---|---|---|
| 30 min | 40 min | 83% |
| 60 min | 70 min | 71% |
| 120 min | 130 min | 46% |
| 200 min | 210 min | 13% |
If review takes 200 minutes, almost all of the benefit is gone. Two vendors with the same generation speed can have very different returns, depending on how hard their output is to check.
Why reviewing GenAI output is hard
The difficulty is that GenAI output is fluent. Errors do not look like errors. In my talk I walked through six ways the output goes wrong, adapted from research on the failure points of retrieval-based AI systems (Barnett et al., CAIN 2024):
- Missing content: something the document needs is simply not there.
- Retrieval gaps: the AI did not find the relevant source in the first place.
- Extraction failure: the source was found, but the value was pulled out wrongly.
- Wrong specificity: the answer is too general, or too narrow, for the question.
- Incomplete answers: part of the answer is present and part is not.
- Inter-context conflict: the source material contradicts itself, and the output hides it.
A reviewer who does not know which of these to look for reads every line with equal suspicion, and that is where the minutes go.
What to ask any vendor
Before you accept an ROI claim, ask for five numbers, not one:
- Time to generate.
- Time to review, measured by your own reviewers on your own documents.
- Rework rate: how often output has to be corrected or regenerated.
- Error catch rate: how many planted or known errors reviewers actually find.
- Total time per approved document.
Run a small pilot and measure all five. It takes a few weeks and it replaces a claim with evidence.
How AI-assisted human review closes the gap
The answer is not to remove the human. In GxP the human decision is the control, and an auditor will expect to see recorded evidence of it. The answer is to make the human review faster and more reliable, by designing it into the platform instead of leaving it to each reviewer.
This is the approach we take at CIMCON with AI-assisted human-in-the-loop review:
- Ready-made agents. Users do not write their own prompts, so output is consistent and one validated set of prompts means something.
- Traceability to the source. Each output links back to the evidence it came from, so the reviewer checks the claim instead of searching for it.
- Errors logged by type. AI errors are recorded against a common error taxonomy, which tells reviewers what to look for and tells you where the AI is weak.
- The review is the record. Reviewer, decision and evidence sit in one audit trail, alongside the rest of your validation records.
If review time drops from 120 minutes to 60, the saving in Table 1 moves from 46% to 71%. That is an assumption for illustration, not a measured result, but it shows where the value is. The biggest lever on ROI is not a faster draft. It is a faster, better-supported review.
Table 3: The effect of a better-supported review (illustrative)
| Review without aids | Review with traceability and logged errors | |
|---|---|---|
| Time to generate | 10 min | 10 min |
| Time to review | 120 min | 60 min |
| Total per document | 130 min | 70 min |
| Saving against manual | 46% | 71% |
The takeaway
When you evaluate GenAI, do not stop at how fast it writes. Measure how long it takes a qualified person to trust the result. That is the cost that decides your return, and it is the one an auditor will ask you to show evidence for.
| Want to see how this works in practice, or to design a pilot that measures all five numbers? Contact CIMCON.
The examples in this post are illustrative and use assumed numbers. They are not CIMCON customer results. |
