Test automation in a validated environment

What changes when the system under test is already validated?

The pipeline stops being an observer of the system and becomes a user of it. FDA defines computer software assurance as "a risk-based approach for establishing and maintaining confidence that software is fit for its intended use", and says the approach also establishes and maintains that software is "in a state of control throughout its life cycle", which the guidance calls the validated state. A test run that logs in, creates a patient record, edits it and leaves it there has acted inside that state of control. The run is now something the system's own history has to account for.

Three consequences follow, and none of them is about test coverage. The run creates electronic records under 21 CFR 11.3(b)(6), which defines an electronic record as any combination of text, graphics, data, audio, pictorial or other information representation in digital form created, modified, maintained, archived, retrieved or distributed by a computer system. The run acts under an identity, and section 11.10(g) requires authority checks ensuring that "only authorized individuals can use the system, electronically sign a record, access the operation or computer system input or output device, alter a record, or perform the operation at hand". The run leaves entries in the system's audit trail, which section 11.10(e) requires to "independently record the date and time of operator entries and actions that create, modify, or delete electronic records".

Whether the destination counts as a closed system turns on one thing. 11.3(b)(4) defines a closed system as an environment in which system access is controlled by persons responsible for the content of the electronic records on it, and 11.3(b)(9) defines an open system as one where it is not. Hosting, internet exposure and cloud tenancy appear nowhere in either definition. A hosted runner holding credentials to your validated system is therefore a question about who controls access, and the answer decides whether 11.30 adds encryption and digital signature measures on top of everything in 11.10. The eleven controls that section lists, and which of them FDA says it enforces, are set out at what 21 CFR Part 11 requires of a closed system.

Test environments that hold patient data raise a separate question before any of this. We do not need production PHI to test. Environments run on synthetic and de-identified data. The terms we work under, and the two questions on that subject this site publishes as still unanswered, are set out at how we work with protected health information.

Which of your pipeline's outputs are regulated records?

The ones a predicate rule already requires you to keep. 21 CFR 11.1(b) applies Part 11 to records in electronic form created, modified, maintained, archived, retrieved or transmitted under records requirements set out in agency regulations. The 2003 Scope and Application guidance states that records not required to be retained under predicate rules, but nonetheless maintained in electronic format, are not Part 11 records. The assurance guidance applies the same test from the other end: to decide when a record is required, consider whether it would be necessary as evidence to document required validation.

That test cuts a pipeline in two, and the cut does not follow the directory structure.

Outputs of a test automation pipeline sorted by whether a predicate rule stands behind them, and what follows for each under 21 CFR Part 11.

Pipeline output	Is a predicate rule behind it?	What follows
Console log of a developer's local run, read once and discarded	No	Nothing. It is not a record because no rule asks for it
The test report filed as verification evidence for a release	Yes, the validation obligation the release rests on	Held electronically, it is generally an electronic record
Routine activity logs a commercial tool writes at startup	No, where they are not evidence of the validated state	The guidance's own example of a log outside Part 11
Data the run wrote into the validated system itself	Whatever rule governs that data	The system's controls apply, and the run is inside them

The guidance gives the paired examples directly: documentation showing that an enterprise system correctly and reliably automates checking materials before use in production is necessary as evidence of a validated state, so Part 11 generally applies to it when it is held electronically, while commercial off-the-shelf routine activity logs saved automatically at application startup, not necessary as evidence of a validated state, fall outside. Which category each of your artefact stores lands in is a decision to write down once, and to write down before the retention policy is set.

Can an automated run sign a record?

No, and the regulation closes the route three separate times. 11.100(a) requires each electronic signature to be "unique to one individual" and states it "shall not be reused by, or reassigned to, anyone else". 11.200(a)(2) requires signatures to be used only by their genuine owners. 11.200(a)(3) requires signatures to be administered and executed so that "attempted use of an individual's electronic signature by anyone other than its genuine owner requires collaboration of two or more individuals". A signing credential placed in a pipeline secret store is available to everyone who can read that store, which is the condition all three provisions exist to prevent.

The practical rule is that automation drives the workflow up to the signature and a person executes it. That boundary belongs in the test plan, because a suite written without it either fakes the signature or stores a real one.

section 11.200(a)(1) then creates a branch that user interface automation reliably misses. Within "a single, continuous period of controlled system access", the first signing uses all electronic signature components and subsequent signings use at least one component executable only by the individual. Signings outside such a period each use all components. A suite that authenticates once in a setup step and then signs twelve records exercises only the second branch, twelve times. Nothing in the report says the first branch was never entered. Building the session boundary into the test data is the subject of electronic signature testing under 21 CFR Part 11; what the pipeline owes here is a run configuration that can produce both cases on purpose.

In what order do you qualify a pipeline?

The order matters because step 3 sizes every step after it. A team that starts at step 5 has either bought effort it cannot justify or skipped effort it cannot defend.

Step 1 is per job and not per repository, because the assurance guidance does the analysis per feature, function or operation and not per system. It states that software may have more than one intended use depending on its individual features, functions and operations, which may present different risks and different levels of validation effort, and that where all features share one intended use the manufacturer may assess the software overall. FDA recommends documenting the decision-making process for determining whether a feature, function or operation is used as part of production or the quality management system. A monorepo holding a smoke suite, a performance harness and a migration rehearsal holds three intended uses and one commit history.

Step 3 uses a risk question that is not the device risk question. The assurance guidance says its analysis is distinct from a device risk analysis under ISO 14971:2019, and focuses instead on factors that may impact or prevent the software from performing as intended: proper system configuration and management, security of the system, data integrity, data storage, data transfer, and operation error. It asks for failures that are reasonably foreseeable as opposed to likely, and offers a power outage as its example. Where the sort itself lands for each category of software, and the six elements the record contains, are set out at FDA computer software assurance.

What does step 4 actually buy you?

Vendor evidence, in place of testing you would otherwise fund yourself. The guidance lists established purchasing controls for selecting and monitoring software vendors as a leverageable control, and says the vendor's own development practices and validation work are a starting point, which for some lower-risk items may be all the assurance needed. It names what to look at.

Vendor evidence FDA's computer software assurance guidance lists for review, from onsite audit to data integrity capabilities, with what the guidance says of each.

What you review	What the guidance says about it
Onsite audit of the vendor	Named first, with the acknowledgement that auditing the vendor may not be feasible or appropriate
Accreditations and certifications	Service Organization Controls reports and ISO certifications are the two examples given
Development and quality assurance practices	The vendor's documentation for software development and software quality assurance
Cybersecurity documentation	Security risk assessments, threat modeling, security design and development reviews, SBOM, testing
Data-integrity capabilities	Record retention and archiving, securing data at rest and in transit with secure computer-generated time-stamped audit trails and encryption, access controls, electronic signature controls, authorization checks

For an electronic system used in a clinical investigation, FDA's October 2024 questions and answers guidance is more specific about the division of labour. Where an information technology service provider performs the validation, the regulated entity deploying the system may review the provider's processes for developing and managing the system, its validation processes, its functional testing of the system, and its change control procedures and tracking logs. The same guidance adds the part that decides who is exposed: FDA may request documentation of system validation during an inspection, and it is the responsibility of the regulated entity to ensure that documentation is available if requested, including documentation created and maintained by the service provider. A contract with a hosted test runner that gives you no right to obtain that documentation has moved the evidence outside your reach and left the obligation inside it.

Who may change a test, and how does the change get recorded?

Whoever the authority checks allow, and the answer has to be written before it is asked. 11.10(g) enumerates five privileges and login is one of them: use the system, sign a record, access the operation or an input or output device, alter a record, and perform the operation at hand. section 11.10(d) limits system access to authorized individuals. Where the test protocol itself is documentation for system operation and maintenance, 11.10(k)(1) requires adequate controls over its distribution, access and use, and 11.10(k)(2) requires "revision and change control procedures to maintain an audit trail that documents time-sequenced development and modification of systems documentation".

A version control system supplies part of that and not all of it. It records who committed, when, and what changed. It does not record who was authorised to make that class of change, and a repository where every engineer can rewrite an assertion in a filed protocol has answered 11.10(k)(2) and left 11.10(g) open. The gap closes with a branch rule that names approvers for the protected paths, and with the approval recorded where the record is kept rather than only in the hosting provider's interface.

section 11.10(i) is about the people who touch the system. It requires a determination that persons who develop, maintain or use electronic record and signature systems have "the education, training, and experience to perform their assigned tasks". The 2024 clinical investigations guidance footnotes that same provision when it lists records related to staff training on the use of electronic systems among the things an inspection covers for a clinical investigator, alongside the requirement that sponsors are notified of personnel changes so access rights can be revoked. A contractor who wrote a migration harness in March and whose account still resolves in September is inside both of those sentences.

What the run record itself has to name is separate again, and IEC 62304 sets it: clause 5.7.5 lists the relevant test tools and the identity of the person responsible for executing the test among its seven required items, at every software safety class. IEC 62304 software testing requirements covers which clauses bind at which class, and the artefact side of the same problem is worked through in test automation for healthcare products.

What happens when the tooling updates?

The record you wrote in step 6 now describes a different instrument. FDA's 2024 guidance on electronic systems in clinical investigations states the change obligation plainly: changes including software upgrades, security and performance patches, equipment or component replacements and new instrumentation should be evaluated and validated throughout the life cycle of the system depending on risk, should not adversely affect the traceability, authenticity or integrity of new or existing data, and all changes should be documented. The definition of validation the same guidance works from runs "from design until decommissioning of the system or transitioning to a new system". Retiring a test framework is inside that span, so the migration off it is a documented life cycle event and not a cleanup task.

Three kinds of update behave differently and a single policy for all three is wrong for at least two.

• A pinned dependency you bump deliberately is a change you can schedule an evaluation around. Its risk is that the evaluation is skipped because the version number moved by one digit.
• A floating dependency resolved at install time changes without a commit. Two runs a month apart ran different code and no record of either says so, which makes step 7 the load-bearing step of the whole sequence.
• A hosted runner the vendor upgrades on their schedule changes underneath you. This is the case the vendor evidence in step 4 has to cover in advance, because the notification may arrive after the run.

For a device with an approved premarket approval or humanitarian device exemption, a change to production or quality management system software also carries a reporting consequence, and the assurance guidance sets out where the line between a 30-day notice and the annual report falls. Which runs a given release reopens is a separate decision with its own method, worked through at regression testing for regulated software.

Which failures does a green run hide inside a validated system?

The ones where the automation took a route the requirement does not live on. Each of these produces a passing report and leaves a control untested.

• Sequencing checks the run never reaches. The requirement at section 11.10(f) is for "operational system checks to enforce permitted sequencing of steps and events, as appropriate". A suite that posts directly to an endpoint has asserted the endpoint. The sequencing control usually sits on the path the suite skipped.
• Copies proved in one form only. section 11.10(b) names two forms of copy, human readable and electronic, and both have to be accurate and complete. An export test that diffs a JSON payload has demonstrated one of them.
• Attribution lost to a shared identity. 11.10(e) requires the audit trail to independently record operator entries and actions. Where every automated action arrives as one service account, the trail records that the account acted and cannot record which run, which build or which person stood behind it.
• Signature meaning asserted as present rather than as correct. 11.50(a) requires the signed record to indicate the printed name of the signer, the date and time of execution, and the meaning associated with the signature, such as review, approval, responsibility or authorship. A check that a signature block exists passes on a record whose meaning field is wrong.
• Controls no software can exercise. 11.300(e) puts a periodic testing duty on the physical devices that carry or generate an identification code or password. Those sit outside any pipeline, and a suite that reports full coverage of the authentication path has not touched them.
• Audit trail content never read back. The trail is a record with its own requirements, and asserting that a row appeared is not the same as asserting what the row says. Audit trail testing requirements covers what to read back and when.

A green pipeline is an assertion that the checks it ran passed. Turning it into an assertion about the system requires the list of checks it never ran, written by someone who read the controls rather than the report.

The clause numbers, quoted wording and guidance dates on this page were checked against the primary text of each document on 2 September 2026: 21 CFR Part 11 as current in the eCFR on 31 August 2026, the Part 11 Scope and Application guidance of August 2003, the Electronic Systems, Electronic Records, and Electronic Signatures in Clinical Investigations questions and answers guidance of October 2024, the Computer Software Assurance for Production and Quality Management System Software guidance issued 3 February 2026, which is final and nonbinding, and IEC 62304 Edition 1.1.