Join Essenvia at RAPS Convergence 2026 — Booth #918
· Charlotte, 15–17 September
Read more →
Validating AI tools for regulatory work: a practical governance framework
The regulatory operating model
By Soumya Mahapatra
Your regulatory team is already using general-purpose assistants on regulatory work for weeks, unlogged and uncovered by any SOP. Elsewhere, procurement is qualifying an AI tool and will hand you a questionnaire to sign. Both ask the same question: what governs the use of AI?
Validate the use case, not the tool
Computerized system validation assumes a stable relationship between software and what it does: you qualify the system, control changes, and the qualified state holds until you change something yourself.
AI breaks that in both directions. The same tool, unchanged, is trivial when it reformats a table and consequential when it advises on whether a design change is significant. And it changes under you, on someone else's schedule, without any action on your part.
So the unit of governance is the use case: a stated input, a stated output, a place the output lands, a named control, a named owner. One tool carries five use cases and five decisions. If you cannot write the intended use in one sentence naming what goes in, what comes out and what it is used for, you have approved a vendor rather than scoped a use case.
Classify by consequence and detectability
Most risk frameworks stop at consequence: where the output lands, what happens if it is wrong. Necessary, not sufficient.
The characteristic failure of a language model is not a blank output or an error message. It is a fluent, confidently wrong answer that reads exactly like a correct one. Against that, the question deciding whether a control is real is detectability: can the reviewer tell the output is wrong, from an independent source, within the time budgeted for review?
A high-consequence task checkable against a clause in ten minutes is safer than a low-consequence task nobody can check. Rank on both axes and the tiers fall out.
Tier | Where the output lands | Control that must exist |
1 · Orientation | Stays with the person who generated it. Never quoted, never filed. | Competence and a confidentiality rule. No qualification record. |
2 · Draft material | Becomes text in a document a competent human verifies line by line against primary sources. | Mandatory source check before release. The reviewer must hold the sources. |
3 · Decision support | Informs a regulatory determination — change significance, classification, pathway, comparability. | An independent human determination that stands on its own record. Qualification against a gold set. |
4 · Record or action | Enters a controlled record or submission, or the tool writes to a system. | Full validated state under CSV/CSA procedure. Where the regulation requiring the record, not Part 11 — applies: validation, audit trail, authority checks, signature manifestation, record-signature linking (21 CFR 11). |
Tier 4 is where most pilots quietly end up, and where almost no pilot is qualified to be. Tier 3 holds the interesting judgment, clustering around decisions that are expensive to reverse — like whether a device change requires a new submission or the registration ownership choice in a new market.
Figure: Decision tree routing a proposed AI use case through four questions to one of four risk tiers or to a not-approvable outcome.
What "human in the lead" actually requires
"Human in the loop" is usually asserted rather than designed. Test any claimed human control with one question: what would the reviewer have to do to catch this model's characteristic failure? If the answer is "read it carefully," there is no control. Reading a fluent wrong sentence carefully returns a fluent wrong sentence.
Three things, in practice.
The reviewer holds the source and is expected to open it. The signature attests to a source check, not to having read the output. Write it that way and people behave differently.
The reviewer could have produced the output themselves. Someone who could not have written the answer cannot judge it — which is why junior-analyst use cases belong a tier lower than they appear.
The reviewer sees the question. A leading prompt produces a confirming answer, and only the prompt shows it. Retain prompts from Tier 3 up.
Which is why the most common sentence in AI tool documentation — all AI outputs undergo mandatory human review before finalization — is not a control. It describes your staff's behavior, not the tool's, and no vendor can enforce or evidence it. It transfers your obligation rather than discharging it. So does a review budgeted at less time than the task would take manually.
Two kinds of evidence
For any tool at any tier, evidence divides in one place: what you must generate and what you must obtain. No provider can generate the first for you; you cannot generate the second for yourself. Programs fail in the second half, because a completed questionnaire looks like it has answered it — when usually it is a security questionnaire with "AI" prepended: necessary, and no evidence the output is correct.
What you generate.
Artifact | What it must contain |
Intended use statement | One sentence, scoped so a different task is visibly a different use case. |
Risk classification | The tier, with the reasoning for the detectability call written down. |
Testable acceptance criteria | A gold set: 20–50 items from your real work, correct answers set independently by a qualified human, threshold expressed in terms of the control. For Tier 3: every wrong answer must be one the reviewer's stated check catches. Count confidently-wrong outputs separately from misses |
Run records | Date, model and version, settings, prompt, output, pass/fail per item, who adjudicated. Running the gold set more than once — materially different determinations across runs is a finding, not noise. |
Input-side security review | Validation asks whether the output is correct, not whether the input can be manipulated. Review against the OWASP Top 10 for LLM applications. Prompt injection through ingested documents dominates: supplier files and portal downloads carry text a model may follow as instruction. |
What you obtain. Ask these of a procured platform, and of the general-purpose assistant your team already uses — for the latter you get the answers by reading the terms and the model documentation. , not by asking anyone. The weak answers share one property: each is probably true, and none of it is checkable.
Ask | Weak response | What counts as evidence |
Which models, at what version, and who decides when they change? | A list of model provider names. | Named models and versions, pinned or floating, the deprecation notice period, and whether you hear before a change or after it. |
What leaves our environment, and does it train anything? | "Encrypted in transit and at rest." "We don't train on customer data." | Whether whole documents or fragments reach third-party model providers, what is masked first, and retention and training terms covering every subprocessor — not only the tool you contracted with. |
How was accuracy measured, on what, and can we see it? | "Validated for accuracy prior to deployment." | Composition of the evaluation set by jurisdiction and document type, who set ground truth, the metric, the threshold, the result. A set that can be inspected. |
What traceability comes with each output? | "Outputs are explainable." "Confidence scores have surfaced." | Per-output logs of model version, prompt, retrieved sources and timestamp — exportable, available. Traceability to a retrieved passage shows what grounded an answer; and confidence scores checked against measured accuracy. |
The alternative to freeze-and-requalify
Traditional validation treats change as the enemy: qualify, freeze, re-qualify when something moves. Point that at a hosted model re-pointed on someone else's schedule and you get a program either permanently out of date or permanently re-testing.
FDA has already worked this problem on the device side, and the structure is worth “borrowing”. A predetermined change control plan sets out in advance what changes you intend to make, how each will be verified, and what happens when verification fails — so an anticipated change is implemented inside the plan instead of triggering a fresh qualification ("Predetermined Change Control Plans for Medical Devices," draft guidance for industry and FDA staff, CDRH and CBER, issued August 22, 2024).
Three components, that are critical for this context:
Description of changes. The specific changes you anticipate — not a blanket reservation of the right to change anything. A short enumerated list, each item specific enough to test.
Change protocol. For each one, the verification method and acceptance criteria, fixed before the change. Plus the commitment that if criteria are not met, the failure is recorded and the change is not implemented.
Impact assessment. The risk of each change alone, how one interacts with another, and the cumulative effect of all of them.
The routing test is the part most worth stealing, because it is not whether this change is big or small. It is whether the change introduces a new risk or modifies a risk you have already characterized. A change that moves the severity or likelihood of a hazard your controls were built for can sit inside the plan. One that introduces a hazard those controls were never designed to catch cannot, and needs the full exercise.
Two boundaries follow. A change to what the use case is for is a new intended use, not a modification. And a change made to fix a failure is out of scope, because no plan pre-authorizes remediation of a problem it did not anticipate.
Translated for your gold dataset: a model version change is anticipated and pre-specified, verified by the gold set against a threshold that suspends the use case if missed. A change of task is not a version change, however small the prompt edit looks.
Scale the asking to the tier
Running all four obtain questions on a Tier 1 use case will stall diligence and your team will route around it. Tiers 1 and 2 need the data-handling row, nothing more. Tier 3 adds accuracy, evidence and traceability. Tier 4 needs all four as contract commitments rather than questionnaire answers, plus one more: what deterministic, non-model check runs on the output, and what it does not cover.
Run this against a consumer-tier assistant and the accuracy and traceability rows come back empty. That is not a gap to escalate — it is the answer because those products do not publish that material. The empty rows are the finding: that tool is capped at Tier 2. Write that ceiling into your procedure rather than banning the tool; bans push underground use.
Three other frameworks bear on this, and all three need a currency check before you cite them: ISPE's GAMP 5 second edition, FDA's computer software assurance guidance, and FDA's credibility-assessment framework for AI in regulatory decision-making.
An auditor’s ask and responses
Three questions:
Is this system validated? — should be answered with scope: the use case, the tier, and the qualification record for that task.
What happened when the model changed? — the change plan, the verification run, the result, the decision.
Show me a case where the tool was wrong — you should have one, with what you did about it.
Start here Monday
Open a spreadsheet. One row per AI use case happening in your group — ask people, because the unlogged ones are the point. Four columns: what goes in, what comes out, where it lands, who signs. Assign a tier.
Anything at Tier 3 or 4 without a named signer stops until it has one. Then build the gold set for your highest-volume Tier 3 case: twenty items, answers set by a human, run and recorded. Write the change plan in the same sitting — the version you tested, the changes you expect, the threshold that stops the use case.
Your regulatory team is already using general-purpose assistants on regulatory work for weeks, unlogged and uncovered by any SOP. Elsewhere, procurement is qualifying an AI tool and will hand you a questionnaire to sign. Both ask the same question: what governs the use of AI?
Validate the use case, not the tool
Computerized system validation assumes a stable relationship between software and what it does: you qualify the system, control changes, and the qualified state holds until you change something yourself.
AI breaks that in both directions. The same tool, unchanged, is trivial when it reformats a table and consequential when it advises on whether a design change is significant. And it changes under you, on someone else's schedule, without any action on your part.
So the unit of governance is the use case: a stated input, a stated output, a place the output lands, a named control, a named owner. One tool carries five use cases and five decisions. If you cannot write the intended use in one sentence naming what goes in, what comes out and what it is used for, you have approved a vendor rather than scoped a use case.
Classify by consequence and detectability
Most risk frameworks stop at consequence: where the output lands, what happens if it is wrong. Necessary, not sufficient.
The characteristic failure of a language model is not a blank output or an error message. It is a fluent, confidently wrong answer that reads exactly like a correct one. Against that, the question deciding whether a control is real is detectability: can the reviewer tell the output is wrong, from an independent source, within the time budgeted for review?
A high-consequence task checkable against a clause in ten minutes is safer than a low-consequence task nobody can check. Rank on both axes and the tiers fall out.
Tier | Where the output lands | Control that must exist |
1 · Orientation | Stays with the person who generated it. Never quoted, never filed. | Competence and a confidentiality rule. No qualification record. |
2 · Draft material | Becomes text in a document a competent human verifies line by line against primary sources. | Mandatory source check before release. The reviewer must hold the sources. |
3 · Decision support | Informs a regulatory determination — change significance, classification, pathway, comparability. | An independent human determination that stands on its own record. Qualification against a gold set. |
4 · Record or action | Enters a controlled record or submission, or the tool writes to a system. | Full validated state under CSV/CSA procedure. Where the regulation requiring the record, not Part 11 — applies: validation, audit trail, authority checks, signature manifestation, record-signature linking (21 CFR 11). |
Tier 4 is where most pilots quietly end up, and where almost no pilot is qualified to be. Tier 3 holds the interesting judgment, clustering around decisions that are expensive to reverse — like whether a device change requires a new submission or the registration ownership choice in a new market.
Figure: Decision tree routing a proposed AI use case through four questions to one of four risk tiers or to a not-approvable outcome.
What "human in the lead" actually requires
"Human in the loop" is usually asserted rather than designed. Test any claimed human control with one question: what would the reviewer have to do to catch this model's characteristic failure? If the answer is "read it carefully," there is no control. Reading a fluent wrong sentence carefully returns a fluent wrong sentence.
Three things, in practice.
The reviewer holds the source and is expected to open it. The signature attests to a source check, not to having read the output. Write it that way and people behave differently.
The reviewer could have produced the output themselves. Someone who could not have written the answer cannot judge it — which is why junior-analyst use cases belong a tier lower than they appear.
The reviewer sees the question. A leading prompt produces a confirming answer, and only the prompt shows it. Retain prompts from Tier 3 up.
Which is why the most common sentence in AI tool documentation — all AI outputs undergo mandatory human review before finalization — is not a control. It describes your staff's behavior, not the tool's, and no vendor can enforce or evidence it. It transfers your obligation rather than discharging it. So does a review budgeted at less time than the task would take manually.
Two kinds of evidence
For any tool at any tier, evidence divides in one place: what you must generate and what you must obtain. No provider can generate the first for you; you cannot generate the second for yourself. Programs fail in the second half, because a completed questionnaire looks like it has answered it — when usually it is a security questionnaire with "AI" prepended: necessary, and no evidence the output is correct.
What you generate.
Artifact | What it must contain |
Intended use statement | One sentence, scoped so a different task is visibly a different use case. |
Risk classification | The tier, with the reasoning for the detectability call written down. |
Testable acceptance criteria | A gold set: 20–50 items from your real work, correct answers set independently by a qualified human, threshold expressed in terms of the control. For Tier 3: every wrong answer must be one the reviewer's stated check catches. Count confidently-wrong outputs separately from misses |
Run records | Date, model and version, settings, prompt, output, pass/fail per item, who adjudicated. Running the gold set more than once — materially different determinations across runs is a finding, not noise. |
Input-side security review | Validation asks whether the output is correct, not whether the input can be manipulated. Review against the OWASP Top 10 for LLM applications. Prompt injection through ingested documents dominates: supplier files and portal downloads carry text a model may follow as instruction. |
What you obtain. Ask these of a procured platform, and of the general-purpose assistant your team already uses — for the latter you get the answers by reading the terms and the model documentation. , not by asking anyone. The weak answers share one property: each is probably true, and none of it is checkable.
Ask | Weak response | What counts as evidence |
Which models, at what version, and who decides when they change? | A list of model provider names. | Named models and versions, pinned or floating, the deprecation notice period, and whether you hear before a change or after it. |
What leaves our environment, and does it train anything? | "Encrypted in transit and at rest." "We don't train on customer data." | Whether whole documents or fragments reach third-party model providers, what is masked first, and retention and training terms covering every subprocessor — not only the tool you contracted with. |
How was accuracy measured, on what, and can we see it? | "Validated for accuracy prior to deployment." | Composition of the evaluation set by jurisdiction and document type, who set ground truth, the metric, the threshold, the result. A set that can be inspected. |
What traceability comes with each output? | "Outputs are explainable." "Confidence scores have surfaced." | Per-output logs of model version, prompt, retrieved sources and timestamp — exportable, available. Traceability to a retrieved passage shows what grounded an answer; and confidence scores checked against measured accuracy. |
The alternative to freeze-and-requalify
Traditional validation treats change as the enemy: qualify, freeze, re-qualify when something moves. Point that at a hosted model re-pointed on someone else's schedule and you get a program either permanently out of date or permanently re-testing.
FDA has already worked this problem on the device side, and the structure is worth “borrowing”. A predetermined change control plan sets out in advance what changes you intend to make, how each will be verified, and what happens when verification fails — so an anticipated change is implemented inside the plan instead of triggering a fresh qualification ("Predetermined Change Control Plans for Medical Devices," draft guidance for industry and FDA staff, CDRH and CBER, issued August 22, 2024).
Three components, that are critical for this context:
Description of changes. The specific changes you anticipate — not a blanket reservation of the right to change anything. A short enumerated list, each item specific enough to test.
Change protocol. For each one, the verification method and acceptance criteria, fixed before the change. Plus the commitment that if criteria are not met, the failure is recorded and the change is not implemented.
Impact assessment. The risk of each change alone, how one interacts with another, and the cumulative effect of all of them.
The routing test is the part most worth stealing, because it is not whether this change is big or small. It is whether the change introduces a new risk or modifies a risk you have already characterized. A change that moves the severity or likelihood of a hazard your controls were built for can sit inside the plan. One that introduces a hazard those controls were never designed to catch cannot, and needs the full exercise.
Two boundaries follow. A change to what the use case is for is a new intended use, not a modification. And a change made to fix a failure is out of scope, because no plan pre-authorizes remediation of a problem it did not anticipate.
Translated for your gold dataset: a model version change is anticipated and pre-specified, verified by the gold set against a threshold that suspends the use case if missed. A change of task is not a version change, however small the prompt edit looks.
Scale the asking to the tier
Running all four obtain questions on a Tier 1 use case will stall diligence and your team will route around it. Tiers 1 and 2 need the data-handling row, nothing more. Tier 3 adds accuracy, evidence and traceability. Tier 4 needs all four as contract commitments rather than questionnaire answers, plus one more: what deterministic, non-model check runs on the output, and what it does not cover.
Run this against a consumer-tier assistant and the accuracy and traceability rows come back empty. That is not a gap to escalate — it is the answer because those products do not publish that material. The empty rows are the finding: that tool is capped at Tier 2. Write that ceiling into your procedure rather than banning the tool; bans push underground use.
Three other frameworks bear on this, and all three need a currency check before you cite them: ISPE's GAMP 5 second edition, FDA's computer software assurance guidance, and FDA's credibility-assessment framework for AI in regulatory decision-making.
An auditor’s ask and responses
Three questions:
Is this system validated? — should be answered with scope: the use case, the tier, and the qualification record for that task.
What happened when the model changed? — the change plan, the verification run, the result, the decision.
Show me a case where the tool was wrong — you should have one, with what you did about it.
Start here Monday
Open a spreadsheet. One row per AI use case happening in your group — ask people, because the unlogged ones are the point. Four columns: what goes in, what comes out, where it lands, who signs. Assign a tier.
Anything at Tier 3 or 4 without a named signer stops until it has one. Then build the gold set for your highest-volume Tier 3 case: twenty items, answers set by a human, run and recorded. Write the change plan in the same sitting — the version you tested, the changes you expect, the threshold that stops the use case.
See what your portfolio looks like when everything is sellable — and audit-ready.
A 30-minute demo, on your products and your markets. No sandbox, no trial to configure — a working conversation with people who know regulatory.
See what your portfolio looks like when everything is sellable — and audit-ready.
A 30-minute demo, on your products and your markets. No sandbox, no trial to configure — a working conversation with people who know regulatory.
See what your portfolio looks like when everything is sellable — and audit-ready.
A 30-minute demo, on your products and your markets. No sandbox, no trial to configure — a working conversation with people who know regulatory.