Healthcare test automation for regulated medical software

Why does a test suite need validating before its results count?

Because the suite is the instrument that produced the reading, and a file holding the reading with nothing about the instrument has a gap a reviewer can name. FDA's computer software assurance guidance sorts software by how it is used, and Section V.A(1) places development tools that test or monitor software systems or automate testing activities in the category of software used to support production or the quality management system. Software in that category is still validated under ISO 13485 subclauses 4.1.6, 7.5.6 or 7.6 as appropriate, with the effort often reduced because the risk is lower. The wider sort, including what falls outside the requirement entirely, is at FDA computer software assurance.

A passing run is one program's assertion about another program. It is worth what the first program is worth, and the result on its own gives a reader no way to price that. The ways it goes wrong are ordinary. A selector matches nothing and the step passes silently. An assertion compares a value the system produced against a fixture generated from the same system. A test is skipped by a tag nobody reviewed and the report counts it as skipped rather than as absent. A mock answers where the real interface would have refused. Each of those produces the same green as a test that ran and meant something.

Which parts of the suite are in scope, and which are not?

The parts whose output somebody relies on. A helper a developer runs on a laptop and reads on the screen leaves nothing behind. A job whose report is filed as verification evidence for a release produces a quality record, and the record is why the job is in scope. One repository normally holds both, so the decision is taken at the level of the job that files something rather than at the level of the suite. Which jobs those are is a separate question from which parts of the work can be scripted at all, and several requirements in IEC 62304 ask for an evaluation or an inspection that no script closes: what an automated run and a manual one each produce as evidence sets out where the line falls.

The guidance's worked examples reach this case once. Section V.A(2) takes an automated function that runs GUI test scripts in a separate test environment and rates it low process risk rather than high, because its failure would not produce a quality problem that foreseeably compromises safety. That rating decides the size of the assurance effort. It does not remove the item from the inventory, and an item rated low still carries an intended use, a risk determination and a record.

Which products carry this problem?

The ones where a release decision is written down and read by somebody outside the team that made it. For software as a medical device, the automated runs sit inside the regulatory file rather than beside it, so a suite that cannot say which build it exercised leaves a hole in the file an assessor reads. For a record system, the suite normally spans interfaces owned by other organisations, and a run that is green because a stub answered has demonstrated the stub. For a telemedicine platform, the parts that resist automation hardest, the media path and the device permissions, are the parts a nightly regression is most often assumed to have covered.

What does an auditor read instead of the run result?

The record around it. IEC 62304 sets the contents of a software system test record at clause 5.7.5, and states that the purpose is to support the repeatability of tests. Seven items are required: a reference to the test case procedures showing the required actions and the expected results; the result as pass or fail together with a list of anomalies; the version of the software tested; the relevant hardware and software test configurations; the relevant test tools; the date tested; and the identity of the person responsible for executing the test and recording the results. That subclause binds at every software safety class, class A included.

Two of the seven are where a pipeline usually has nothing to offer. The test tools item asks which tool produced the result and at which version, and a job that resolves its dependencies at install time cannot say. The identity item asks who is responsible for executing the test and recording the result, and a scheduled job answers that only if somebody was named for it in advance.

clause 5.7.4 then asks for what a run cannot supply about itself: an evaluation of the appropriateness of the verification strategies and the test procedures, and verification that every software requirement has been tested or otherwise verified, that the traceability between requirements and tests is recorded, and that results meet the pass or fail criteria. A coverage percentage answers none of those. Which clauses bind at which software safety class is set out at IEC 62304 software testing requirements, and the form the trace itself has to take is covered under validation documentation and requirements traceability.

Which requirements does the work touch?

Requirements from ISO 13485:2016, FDA computer software assurance guidance and IEC 62304 that reach an automated test suite, and what each means for it.

What binds	Where it comes from	What it means for an automated suite
Validate software used as part of, or in support of, production or the quality management system	ISO 13485:2016 subclauses 4.1.6, 7.5.6 and 7.6, incorporated into 21 CFR Part 820	The suite gets an intended use, a risk determination and an assurance record of its own
Development tools that automate testing activities count as software used to support the quality management system	FDA computer software assurance guidance, Section V.A(1), nonbinding	The tooling belongs in the inventory, at an effort scaled to its process risk
A system test record names the version tested, the configurations, the test tools, the date and the responsible person	IEC 62304 clause 5.7.5, at every software safety class	The pipeline has to emit those items, because a console log carries none of them
Requirements-to-test traceability recorded, and the verification strategies and test procedures evaluated for appropriateness	IEC 62304 clause 5.7.4, at every software safety class	Somebody reviews what the suite covers, separately from whether the suite passes
Retest after changes	IEC 62304 clause 5.7.3, at every software safety class	A written rule decides which runs a release reopens, before the release

Two of the three ISO 13485 subclauses FDA names for this obligation carry titles that describe what a test rig is: 7.5.6 is titled Validation of processes for production and service provision, and 7.6 is titled Control of monitoring and measuring equipment. Where the same obligation reaches your quality system software rather than your test suite, the protocols and executed records are under FDA software validation.

What do you receive?

The artefacts listed below, one set per job that files a record. Two of them change a plan rather than document one.

The register of what the suite is allowed to close is the first. Some requirements cannot be closed by an assertion at all: an alarm a clinician has to notice, an acceptance that is a clinical judgement, a display whose legibility is the requirement. Writing that split down before the automation is built stops a coverage figure from standing in for evidence it never produced.

The repeated-run log is the second, and it exists because retrying is ordinary engineering and a repeat is a record event. A test that failed and passed on the second attempt produced two results. A file that keeps only the second has removed the anomaly that clause 5.7.5 asks to be listed beside the pass, and the removal leaves no trace in the report.

How does this fit a team that ships every week?

By fixing, before the pipeline is trusted with anything, which changes to the tooling reopen which evidence. Every framework upgrade is a change to the instrument. A runner major version, a browser driver, an assertion library, a change in how the reporter serialises a result: each can alter what a test asserts without altering a line of the test. IEC 62304 requires retesting after changes at clause 5.7.3, at every class. A team with a rule for changes to the product and no rule for changes to the tooling has written half of it.

Running the suite inside a system that is itself under validation adds constraints a pipeline does not have on its own, and those are covered at test automation in a validated environment. Which environment the suite may run against, and on what terms an outside team reaches it, is its own question, raised at working with protected health information.

When does an automated suite stop being evidence?

• The report is generated by the build system and kept in the build system, so it expires under a retention policy nobody set with a quality record in mind.
• Dependencies resolve at install time, so two runs a month apart executed against different library versions and neither record says so.
• A test was disabled behind a tag during a release crunch, and the report counts it as skipped rather than as not run.
• The only surviving evidence of a failure is that the job is green now, because it was restarted.
• Assertions compare the system against a fixture generated from the system, so the test agrees with whatever the system does.
• The environment is rebuilt per run from a definition that keeps changing, so no result can be attributed to a configuration.
• Nobody is named against a scheduled run, and the record has an empty field where clause 5.7.5 puts the person responsible.
• The suite is credited with closing requirements no assertion can reach, because coverage was counted per test rather than per requirement.

The clause numbers and the guidance wording on this page were checked against the primary text of each document on 2 September 2026: IEC 62304 Edition 1.1, and the FDA guidance Computer Software Assurance for Production and Quality Management System Software, issued 3 February 2026, which is final and nonbinding.