How to validate an AI model in clinical software

What does a model validation have to establish that ordinary software testing does not?

Four claims that a deterministic suite never has to argue: which population the numbers came from, why the acceptance threshold is the one it is, which artefact produced each result, and what a clinician did with the output before it reached a patient. Everything else in the test plan is ordinary software testing and is judged by the ordinary rules.

The reason the four appear is that the object under test stopped being a rule someone wrote. Article 3, point (1), of Regulation (EU) 2024/1689 defines an AI system as one that "infers, from the input it receives, how to generate outputs such as predictions, content, recommendations, or decisions". Article 3, point (18), then defines performance of an AI system as "the ability of an AI system to achieve its intended purpose", and the MDR uses the same construction at Article 2(22) for a device, "the ability of a device to achieve its intended purpose as stated by the manufacturer". Both definitions point at the intended purpose statement and not at a specification of behaviour, so an expected result cannot be derived from the code.

That changes what a pass means. A single case still passes or fails, and the claim being evidenced is a rate over a population, so the case on its own carries almost nothing. The Regulation's data vocabulary already recognises this separation. Article 3 defines training data at point (29) as data used "for training an AI system through fitting its learnable parameters", validation data at point (30) as data for evaluating the trained system and tuning its non-learnable parameters, and testing data at point (32) as data "used for providing an independent evaluation of the AI system in order to confirm the expected performance of that system before its placing on the market or putting into service".

Which of these obligations reach your product, and from which date, is set out on the EU AI Act requirements page.

How do you compare the population the model was fitted on with the population it will meet?

By writing both down as characteristics and putting the differences in the risk file. Article 10(3) requires training, validation and testing data sets to have "the appropriate statistical properties, including, where applicable, as regards the persons or groups of persons in relation to whom the high-risk AI system is intended to be used". Article 10(4) adds the setting: data sets "shall take into account, to the extent required by the intended purpose, the characteristics or elements that are particular to the specific geographical, contextual, behavioural or functional setting" the system is intended to be used in.

The reporting side closes the loop. Annex IV, heading 3, requires the technical documentation to describe "the degrees of accuracy for specific persons or groups of persons on which the system is intended to be used and the overall expected level of accuracy in relation to its intended purpose". Article 13(3), point (b)(v), puts the same subgroup performance into the instructions for use "when appropriate", and point (b)(vi) adds "specifications for the input data". Subgroup numbers are therefore a documentation obligation, and a validation that reports one aggregate figure has nothing to enter under either heading.

Three differences between the two populations decide the test design, and each one is written as a characteristic on both sides of the comparison.

Three population differences a clinical AI model validation must cover, with what the evaluation set has to carry for each under the EU AI Act.

What differs	What the evaluation set has to carry
Who the patients are	The subgroups named in the intended purpose, each with enough records to report a rate for that subgroup rather than for the whole set
Where the data came from	The sites, devices and acquisition settings of the deployed population, per the geographical and functional setting in Article 10(4)
What the input looks like on arrival	The input specifications of Article 13(3), point (b)(vi), including the malformed and out-of-distribution inputs a live feed produces

The MDR asks the same question about the technical environment. Annex II, Section 6.1(b), fourth indent, requires the software verification and validation evidence to "address all of the different hardware configurations and, where applicable, operating systems identified in the information supplied by the manufacturer", and it requires the summary results of testing "both in-house and in a simulated or actual user environment prior to final release". A model evaluated only on an archive extract has covered neither the second environment nor the configuration spread. The wider set of MDR software obligations is described on the EU MDR software requirements page.

An evaluation corpus assembled from real encounters is protected health information for as long as it is identifiable, so the agreement covering it is settled before the corpus is copied into any environment. We sign a Business Associate Agreement before any engagement that touches PHI, and the rest of the boundary is written out under how we work with protected health information.

Who owns the threshold, and when does it have to be fixed?

The manufacturer owns it, and it is fixed before the evaluation runs. Article 9(8) states that testing "shall be carried out against prior defined metrics and probabilistic thresholds that are appropriate to the intended purpose of the high-risk AI system". A threshold selected once the distribution is visible fails that condition whatever the resulting number is.

Where the threshold comes from is a risk question, not a modelling one. ISO 14971:2019 states in its Scope that the document "requires manufacturers to establish objective criteria for risk acceptability but does not specify acceptable risk levels". ISO/TS 24971-2:2026 records at subclause 4.2 that ISO 14971 requires top management to "define and document a policy for establishing the criteria for risk acceptability", and at subclause 4.4 that the risk management plan includes "the criteria for risk acceptability" and "the method to evaluate overall residual risk and the criteria for acceptability of the overall residual risk". The person who signs the risk management policy owns the threshold. A data scientist proposes it.

Two limits are worth knowing before the plan is drafted. ISO/TS 24971-2:2026, "Guidance on the application of ISO 14971, Part 2: Machine learning in artificial intelligence", is a Technical Specification whose first edition is dated June 2026, and it states that it "does not apply to MLMD employing large language models (LLM) or generative AI". It also states that it "does not provide a new risk management process, nor does it expand the requirements of ISO 14971". The process it guides is the one described on the ISO 14971 page.

In what order does the validation run?

Step 5 is the step teams skip, and step 6 is the step that fails an assessment quietly. A subgroup rate that was never computed cannot be entered under Annex IV, heading 3, and computing it afterwards means opening the evaluation set a second time with the results already known.

What does a frozen model version mean for the release record?

It means the weights are a configuration item with an identifier, in the same sense as a source file. IEC 62304 Edition 1.1 puts configuration identification at clause 8.1, and clause 5.7.5, SOFTWARE SYSTEM test record contents, rewritten by Amendment 1 for the stated purpose of supporting the repeatability of tests, lists seven items every system test record documents. Three of them decide whether a model result can be attached to a release: the version of the software tested, the relevant hardware and software test configurations, and the relevant test tools. The other four are the reference to the test case procedures, the pass or fail result with its anomalies, the date tested, and the identity of the person responsible for executing the test and recording the results. All five subclauses of 5.7 bind at software safety classes A, B and C, and the class model is explained on the IEC 62304 testing requirements page.

For a model, "the version of the software tested" has to resolve to more than a container tag. Four artefacts move independently and each one changes the output.

• The weights file, by content hash rather than by file name.
• The preprocessing code, including the normalisation constants baked into it.
• The inference runtime and its numeric settings, which sit under "the relevant test tools".
• The reference data the model reads at inference time, where any is used.

Annex IV, heading 1, already asks the technical documentation for "software and firmware versions and update requirements", and Annex IV, point 2(g), asks for "test logs and all test reports dated and signed by the responsible persons". A notebook output with no identified model behind it satisfies neither, and the reconstruction happens under audit conditions instead.

What happens to last release's test results when the model changes?

They stay valid for the version they name and for no other. IEC 62304 subclause 5.7.3, "Retest after changes", was moved by Amendment 1 from classes B and C to classes A, B and C, so the question binds at every class. Subclause 5.7.4 then requires verification that all software requirements have been tested or otherwise verified and that the traceability between requirements and tests is recorded, which is the check a carried-forward result has to survive.

A retrained model is a change in the sense of both regimes even when no line of application code moved. Article 43(4) requires a new conformity assessment on a substantial modification, and Article 3, point (23), defines a substantial modification as a change "which is not foreseen or planned in the initial conformity assessment carried out by the provider and as a result of which the compliance of the AI system with the requirements set out in Chapter III, Section 2 is affected or results in a modification to the intended purpose". The route that keeps a retrain out of that definition runs through Annex IV, point 2(f), a detailed description of pre-determined changes, which Article 43(4) names expressly. Annex IV, heading 6, then requires "a description of relevant changes made by the provider to the system through its lifecycle", so the changes are recorded whichever side of the line they fall.

Systems that keep learning after release carry one further requirement. Article 15(4) states that such systems "shall be developed in such a way as to eliminate or reduce as far as possible the risk of possibly biased outputs influencing input for future operations (feedback loops), and as to ensure that any such feedback loops are duly addressed with appropriate mitigation measures". A feedback loop is only observable across releases, so the monitoring for it is designed at the same time as the retraining trigger. How a change is analysed and which tests it reopens is the subject of regression testing for regulated software.

How do you evidence that a person reviewed the output before it reached a patient?

With a log entry that records what the reviewer was shown, what the reviewer could do, and what the reviewer did. Article 14(1) requires high-risk systems to be designed so that they "can be effectively overseen by natural persons during the period in which they are in use", and Article 14(4) lists five capabilities the assigned person is to be enabled to exercise: understanding the capacities and limitations of the system, remaining aware of the tendency to over-rely on its output, which the Regulation names automation bias, interpreting the output correctly, deciding in a particular situation not to use the system or to disregard, override or reverse its output, and intervening or interrupting through "a 'stop' button or a similar procedure that allows the system to come to a halt in a safe state".

Four of those five are testable as user interface behaviour. The fourth, the decision to disregard or override, is the one that produces evidence after release, because the override is an event and the event can be counted. Article 12(1) requires the system to "technically allow for the automatic recording of events (logs) over the lifetime of the system", and Article 12(2) requires the logging to enable the recording of events relevant to identifying situations that may result in the system presenting a risk or in a substantial modification, and to facilitating post-market monitoring. An override rate that is indistinguishable from zero is a finding about the interface, and the log is what makes it visible.

The reason the clinician sits in the path at all is a limit the risk standard states about itself. The Introduction to ISO 14971:2019 says that the decision to use a device in the context of a particular clinical procedure "requires the residual risks to be balanced against the anticipated benefits of the procedure", that such decisions "are beyond the scope of this document", and that some of them "can be made only by a qualified medical practitioner with knowledge of the state of health of an individual patient or the patient's own opinion". A validation that measures the model and never observes the review step has measured the smaller half of the system, and MDR Annex II, Section 6.1(b), asks for results from "a simulated or actual user environment" for that reason.

What does the validation hand to an assessor?

Artefacts a clinical AI model validation hands to an assessor, each mapped to the EU AI Act provision or IEC 62304 clause it satisfies.

Artefact	What it answers
Intended purpose statement with named patient groups and setting	Supplies the yardstick that Article 10(4) and Article 3, point (18), both refer back to
Evaluation set characterisation	Shows how the set was judged against the statistical properties in Article 10(3)
Approved and dated test plan with metric, operating point and acceptance criterion	Evidences the "prior defined" condition in Article 9(8)
Metric appropriateness argument	Fills Annex IV, heading 4
Performance report by subgroup	Fills Annex IV, heading 3, and Article 13(3), point (b)(v)
Model configuration record	Supplies "the version of the software tested" and the test configuration required by IEC 62304 subclause 5.7.5
Signed and dated test logs and reports	Fills Annex IV, point 2(g)
Robustness and attack test report	Covers the five vulnerability classes named in Article 15(5)
Human oversight test record with the override path exercised	Evidences the capabilities listed in Article 14(4)
Declared accuracy levels and metrics in the instructions for use	Satisfies Article 15(3)

Assembling these into one file that an assessor reads without a guide is the work described at validation documentation, and Article 11(2) requires the AI content to sit inside the single technical documentation set rather than beside it.

Which failures does a strong accuracy figure hide?

• The reported figure covers the whole evaluation set, and no subgroup rate exists for the groups named in the intended purpose.
• The evaluation ran on a split of the training data, which Article 3, point (31), permits as validation data and which point (32) does not accept as an independent evaluation.
• The operating point was chosen from the curve after the run, so the plan cannot evidence the prior defined condition in Article 9(8).
• The acceptance criterion lives in a modelling document and never entered the risk management plan, so nobody with authority over risk acceptability approved it.
• The test record names a container tag, and the weights inside that tag were replaced without a new identifier.
• A retrain shipped as a data update, with no entry under Annex IV, heading 6, and no pre-determined change description at point 2(f).
• Input handling was tested with well-formed records only, although Article 13(3), point (b)(vi), makes the input specification part of what is declared.
• The oversight interface was demonstrated and never exercised, so no record shows a reviewer overriding an output.
• Post-market data is collected for defects alone, while MDR Annex XIV, Part B, Section 6.1(e), requires the post-market clinical follow-up plan to specify methods for "identifying possible systematic misuse or off-label use of the device, with a view to verifying that the intended purpose is correct".

Articles, annex points and quotations from Regulation (EU) 2024/1689 on this page were read on 2 September 2026 from the consolidated text as at 27 July 2026, and the MDR material from the consolidated text of Regulation (EU) 2017/745 as at 1 January 2026. IEC 62304 references are to Edition 1.1, and the ISO 14971 material comes from ISO 14971:2019 and from ISO/TS 24971-2:2026.