FHIR R4 conformance testing requirements

What does a FHIR conformance claim actually cover?

A FHIR conformance claim covers one document and nothing outside it. The R4 Conformance page puts it in a single sentence: "Systems can only claim FHIR Conformance for functionality described in the applicable CapabilityStatement." The claim is also made against an exchange framework rather than against the specification at large, and R4 names three of them: RESTful FHIR, FHIR messaging and FHIR documents.

The same page calls R4 a closed specification. It "defines what is possible and required for those portions of an API or exchange framework that wish to declare themselves as FHIR conformant". A system may add endpoints and services under other names, and "those portions of their interfaces that extend beyond what FHIR explicitly allows cannot be considered or described as 'FHIR conformant'".

Conformance is claimed rather than awarded. The specification names no conformance testing service, and for anything past resource validation it sends the reader to a page outside the specification. HL7 balloted four portions of R4 as ANSI Normative Standards, Infrastructure, Conformance, Patient and Observation, and states on the page that lists them that "the other parts of the specification do not contain requirements necessary for conformance to the formal standard", and that "implementers are generally focusing on conforming to FHIR R4 as a whole rather than the formal standards". The specification declines to promise safety as well: "Conformance with this specification does not provide any guarantee of patient or data safety."

Until the CapabilityStatement, the profile set and the version string are on the table, a statement that a product supports FHIR names no obligation a test can pass or fail. The equivalent question for an estate still running v2 interfaces is answered in HL7 v2 compared with FHIR.

Which parts of R4 are stable enough to build on?

Eleven of the 145 resources in R4 carry the Normative marker. The resource index names them: Binary, Bundle, CapabilityStatement, CodeSystem, Observation, OperationDefinition, OperationOutcome, Parameters, Patient, StructureDefinition and ValueSet. One resource reaches maturity level 4, MessageHeader, and none reaches level 5. The remaining 133 sit between level 0 and level 3, which places them at Draft or Trial Use.

HL7 attaches a consequence to that status in its Version Management Policy: "There are no rules for maintaining any sort of compatibility between versions for content with these statuses." Draft and Trial Use content may change between versions, breaking changes included. Compatibility rules begin only once an artefact reaches Normative, and even there the specification says FHIR's rules "try to enforce forward compatibility" while backward compatibility "is not guaranteed by FHIR".

The resources a clinical integration is actually built out of are in the unprotected group. The R4 resource index publishes a maturity marker against every resource name. Maturity markers in FHIR R4 (4.0.1) for the resources a health product usually touches:

Maturity markers in FHIR R4 (4.0.1) for common health product resources, from Normative for Patient and Observation to FMM 0 for TestReport.

Resource	Maturity marker in FHIR R4 (4.0.1)
Patient, Observation	Normative
Condition, AllergyIntolerance, MedicationRequest	FMM 3
DiagnosticReport, Immunization, Procedure	FMM 3
Practitioner, Organization, DocumentReference	FMM 3
Provenance, AuditEvent	FMM 3
Encounter, Consent	FMM 2
TestScript	FMM 2
ImplementationGuide	FMM 1
TestReport	FMM 0

The levels are cumulative and the criteria are published. Level 2 requires interoperability among at least three independently developed systems using at least 80% of the core data elements, with the results reported to and accepted by the FMG. Level 3 adds a round of formal balloting and at least 10 distinct implementer comments from at least 3 organisations resulting in at least one substantive change. Level 5 requires publication in two release cycles and implementation in at least 5 independent production systems in more than one country. An FMM 3 marker on Condition therefore records that the resource has been balloted and commented on, and records nothing about production deployment.

The marker on a resource is not the whole answer either. R4 states that "some normative resources contain elements labeled as 'trial-use'", giving Bundle.signature as its example, and warns that reading or writing those elements "has the potential to cause breaking change to their applications later". CapabilityStatement is itself mixed: "while the core of the CapabilityStatement resource is Normative, many of the flags that indicate exactly how the system operates are marked as trial-use".

The ANSI Terminology and Conformance package leaves several conformance artefacts out. ImplementationGuide, TestScript, TestReport, SearchParameter, ConceptMap and NamingSystem are not in it and are not normative in R4, so the artefacts that describe a guide and that execute a test are less stable than the resources they test.

Which version of R4 are you being asked to support?

Ask for the fhirVersion string, because R4 covers more than one published artefact. R4 was published in December 2018; the current text is 4.0.1, the October 2019 technical correction. The R4 home page records it as "4.0.1, Oct-30 2019: Corrections to invariants & generated conformance resources, and add ANSI Normative Status Notes", and HL7's own machine-readable publication history carries a single R4 entry, version 4.0.1 dated 30 October 2019, with no 4.0.0 entry at all.

The FMG "reviewed and approved all the changes, and confirmed that there is no substantive impact on the ANSI normative content of FHIR R4", while adding that the first set of changes "should be reviewed for impact on non-normative aspects of implementations". The corrections listed include a change of all code system OIDs to the root 2.16.840.1.113883.4.642.4., the addition of the FHIRPath context variable %rootResource and its use in the ref-1 constraint, and a correction to the binding for FamilyMemberHistory.sex. A system built against 4.0.0 and never revisited is running against different generated conformance resources from one built against 4.0.1.

R4 gives three places where the version can be declared. They are alternatives, not a sequence, and a test plan should say which one the parties are reading:

• The fhirVersion element in the applicable CapabilityStatement, StructureDefinition or ImplementationGuide.
• The fhirVersion parameter on the MIME type that applies to the resource.
• A version specific profile named on the resource itself, in Resource.meta.

Two later releases matter to a roadmap conversation. R4B is version 4.3.0, Trial Use, published 28 May 2022, and HL7 states that implementers who do not use the changed portions "can continue to use either R4 or R4B without any functional difference". Its general changes are the CodeableReference and RatioRange types, the NutritionProduct resource, and refinements to invariants that behaved erroneously with missing data. R5 is version 5.0.0, published 26 March 2023, and it is the entry HL7's package list flags as current. Every R4 page carries a banner saying 5.0.0 supersedes it, alongside a promise that the R4 pages stay at their permanent URL.

What has to be pinned before a test can be written?

The CapabilityStatement comes first, because producing it is the only interaction every server has to support. R4 states that "the only interaction that servers are required to support is the capabilities interaction itself", defines it as GET [base]/metadata, and requires that "applications SHALL return a resource that describes the functionality of the server end-point". Three mode values are defined, for the full statement, the normative portions only, and a TerminologyCapabilities resource, and servers may ignore the parameter and return a CapabilityStatement anyway.

R4 makes five CapabilityStatement elements mandatory: status, date, kind, fhirVersion and format. Those five, plus the profiles the statement points at, are what a suite can be written against, and the order below matters because each step needs the output of the one before it.

CapabilityStatement.kind decides what the document is claiming. R4 defines three values. An instance statement "describes the capabilities of a deployed and configured solution available at a particular access point", and the specification calls it "the type of statement that forms a basis for the testing, certification or commissioning of specific software installations". A capability statement describes a software product independent of any installation and "may be used as a marketing tool by software and system developers". A requirements statement documents a wanted system, and R4 says it "might be used as part of an RFP process" and "can be compared against the solution statements proffered by respondents to an RFP".

The three values are enforced by invariants rather than by convention. R4 attaches ten named constraints to CapabilityStatement, nine of them errors and one a warning. cpb-14 requires that if kind is instance, implementation must be present; cpb-15 requires that if kind is capability, implementation must be absent and software present; cpb-16 requires that if kind is requirements, both are absent. cpb-1 requires at least one of the REST, messaging or document elements, and cpb-2 at least one of description, software or implementation. A vendor statement that describes a product while carrying an implementation block fails cpb-15, and no amount of prose around it fixes that.

The distinction between a profile and an implementation guide is the specification's own, and it decides where validation happens. R4 defines a profile as a set of constraints on a single resource, expressed as a StructureDefinition whose derivation is constraint. It defines an implementation guide as "a coherent and bounded set of adaptations that are published as a single unit", and adds a sentence a test harness has to act on: "Validation occurs within the context of the Implementation Guide." A guide holds capability statements, profiles, extensions, value sets and documentation; a profile holds constraints on one structure. Testing against "a profile" without naming its guide leaves the terminology and the dependent profiles unresolved.

One obligation attaches to supportedProfile specifically. A producer of resources "SHALL" mark resources with profile assertions documenting the profiles they conform to, and a server "SHALL" support searching by the _profile parameter for the declared profiles. R4 carries a Trial-Use Note over that whole section, warning of "uninvestigated issues associated with this use of profiles" and telling implementers to expect changes to features and obligations. Both halves belong in the test plan: the obligation is testable today, and the note is why the assertion is worth re-running after a release. The baseline above is also what a medical device interoperability test plan is scoped from.

What in a profile is actually testable?

A constraint becomes testable as soon as the profile writes it down, and Must Support becomes testable only once someone states what it means inside that profile. R4 flags nothing itself: "no elements are flagged as mustSupport=true as part of the base specification", because the base specification "is intended to be independent of any particular implementation context". When a profile sets the flag it "SHALL also make clear exactly what kind of 'support' is required", and R4 offers storage and retrieval, display or capture through a user interface, appearance in an output report, and use in decision support or calculation as examples of what the author might mean.

Must Support is explicitly separate from cardinality: "it is possible to have an element with a minimum cardinality of '0', but still expect systems to support the element". Its meaning has to be recorded in ElementDefinition.definition, in StructureDefinition.description or in the guide's documentation, and it ratchets one way, since a derived profile may change Must Support from false to true and never back. Until that statement exists, a claim that a system supports the Must Support elements asserts nothing a test can check.

Cardinality carries its own warning. The base specification uses only four values, 0..1, 0.., 1..1 and 1.., and very few base elements have a minimum of 1. R4 then states that "setting an element to a minimum cardinality of 1 does not ensure that valid data will be present; specific FHIRPath constraints are required to ensure that the required data will be present", and that minimum cardinalities "should not be taken as a guide to what elements are expected to be present in any particular use of the resource". A suite that asserts only on the presence of an element is therefore asserting less than the profile author meant, and R4 names FHIRPath constraints as the thing that closes that gap.

Terminology bindings are where a profile turns loose text into a testable rule. R4 defines four strengths. Under a required binding "the concept in this element SHALL be from the specified value set". Under extensible it SHALL come from the value set "if any of the codes within the value set can apply to the concept being communicated", and human review decides when it does not. Preferred encourages and does not oblige, and example is illustrative. On a CodeableConcept under a required binding, at least one Coding SHALL be present and one of its values SHALL come from the value set, and text "is not an acceptable substitute for the required code". A profile may tighten a binding and never loosen it: required derives only required, extensible derives required or extensible, and "whatever the constraining profile does, it cannot make codes valid that are invalid in the base profile".

Slicing is the mechanism behind most profile assertions that look like business rules. R4 defines it as splitting a repeating element into sub-lists with different restrictions, and the element whose values tell the slices apart is the discriminator. Five discriminator types exist: value, exists, pattern, type and profile. The specification warns that the profile discriminator "provides the most power" and "requires the most processing (>1000-fold compared to the others)", and tells implementers to use it only where absolutely required. Two consequences reach a test suite directly. Slice names are never exchanged, so an assertion cannot look for the name "systolic" in a payload and has to assert on the discriminating value instead. And a structure that designates no discriminator is describing content R4 itself calls "very difficult to process", which is where a fixture based suite stops distinguishing one slice from another.

Invariants are the last layer, and the specification does not require anyone to run them: "systems are not required to evaluate the constraints, just as they are not required to check for conformance, or schema validity. However, systems SHOULD always ensure that all resources are valid against all applicable constraints." Each constraint carries a key, a severity and a FHIRPath expression that must evaluate to true. Three severities exist. An Error is a rule all resources must conform to, a Warning is reported without invalidating the resource, and a Guideline is a warning marked with the elementdefinition-bestpractice extension that becomes an error only when the implementation context asks a validator to enforce best practice rules. Two validators can therefore disagree on the same resource without either being wrong, and a report that does not name its guideline setting cannot be compared with the next one. Profiles may add constraints and "cannot alter or remove constraints that are already applied".

Extensions have hard rules that a payload either meets or does not. The url is mandatory, "SHALL be a URL, not a URN", and SHALL be the canonical URL of the StructureDefinition that defines the extension. An extension SHALL carry either a value[x] or sub-extensions and never both, and a value[x] that is present SHALL have content. Anything that cannot safely be ignored SHALL be a modifier extension instead, represented as modifierExtension. For modifier extensions R4 adds that any such extension SHALL appear in the narrative, that applications SHALL ensure unrecognised modifier extensions are absent before processing an element that might carry them, and that an application meeting one it does not understand SHALL refuse to process the resource or warn its users.

Four limits bound what any profile may do: it cannot break the base rules, cannot specify default values or meanings, cannot rename or add elements, and it must be safe to process a resource without knowing the profile. R4 states the reason plainly: "knowledge must be explicit in the instance, not implicit in the profile". A profile that mandates behaviour a reader cannot ignore has to mandate a modifier extension to carry it, and a design that hides the obligation in guide prose has put it somewhere no validator will find it.

What can a validator check, and what can it not?

A validator checks eight categories, and the specification says none of its methods covers all of them. Validation covers structure, cardinality, value domains, Coding and CodeableConcept bindings, invariants, profiles, QuestionnaireResponse against its Questionnaire, and business rules made outside the specification. The page that says so is itself marked Informative with no maturity level, so nothing on it is a conformance requirement.

R4 lists six methods, XML Schema, XML Schema with Schematron, JSON Schema, ShEx, the Validator and the validation operation, and then limits all of them: "note that all these validation methods are incomplete; they can only validate the computable aspects of conformance. There are always additional rules made in narrative that they are not able to check." Schema and Schematron are called the least capable because they are not connected to a terminology server, the Java validator "is only as good as the underlying definitions", and "the final arbiter is human inspection of the content of the resources, and the relevant implementation guides and base specification". The sentence a supplier has to keep in view is shorter: "static testing of resource content is not enough to prove conformance to the specification."

The $validate operation is Normative from 4.0.1 and has a contract a harness commonly misreads. It "returns a 200 OK whether or not the resource is valid", and "a 4xx or 5xx error means that the validation itself could not be performed, and it is unknown whether the resource is valid or not". A harness that asserts on the HTTP status is therefore measuring the wrong thing. The return is a single OperationOutcome, and the operation is documented to indicate expected success only when the outcome lists no errors and a mode was specified. If a profile is nominated and the server cannot validate against it, the server SHALL return an error, and R4 notes that servers supporting $validate "generally will only validate against profiles already registered with the server".

OperationOutcome.issue.severity has four codes and they are not interchangeable. fatal means the action failed and no further checking could be performed. error means the issue is important enough to cause the action to fail. warning means the action can still be performed, suboptimally. information has no relation to the degree of success. Reading a report by issue count collapses the difference: a short list after a fatal is a run that stopped early, and a long list of warnings is a valid resource. Which tools produce which of these outputs, and what each one can be trusted with, is covered in FHIR conformance testing tools.

R4 also cautions against running full validation in production, on three grounds it states itself: the computational cost of terminology validation can impose unacceptable delays, validation can cause the loss of critical health care data when one field has an unexpected value, and it can fail poor historical data that cannot be fixed. Its recommendation is Postel's law, "conservative in its sending behavior, and liberal in its receiving behavior", which puts the strict checking in the test environment and the tolerant behaviour in the running system.

Can the regression suite sit on TestScript?

TestScript is Trial Use in R4, at maturity level 2, and TestReport, the artefact a run produces, is at level 0. The Testing FHIR page that describes how the two execute carries the standards status Draft, which the same specification defines as content that "should only be implemented by the brave or desperate". The specification's own executable test artefact is the least settled thing this page names.

The execution model is specific enough to plan around. A run begins by checking whether the interactions listed in the TestScript metadata capabilities section are supported by the server's conformance resource, so a suite refuses a server whose CapabilityStatement does not match. Every setup operation and assertion must complete successfully before the tests run. Execution halts as soon as an operation or assertion fails, the test ends in failure, and the script moves to the next test. Failures in teardown are ignored. An operation expected to produce an error must be followed immediately by assertions that test for it, or the test fails. Conditional assertions are not supported at all, and the documented workaround is to set warningOnly to true, which turns a conditional check into something that never fails.

A TestScript can validate against named profiles during a run: the profiles section names each StructureDefinition by URI and by an id, and an operation references that id through validateProfileId. Fixtures are the resource instances a run uses, they can be marked autocreate and autodelete, and "all defined fixtures are expected to be required in order for the test script to execute". A fixture set cut from production records carries PHI, which is a data handling question before it is a testing one, and how PHI reaches our test environments and under what terms sets out how that is done.

Is US Core part of FHIR R4?

No. US Core is a separate HL7 implementation guide with its own canonical URL, http://hl7.org/fhir/us/core/ImplementationGuide/hl7.fhir.us.core, and its own version numbering. Its current release is 9.0.0, STU9, standards status Trial-use, maturity level 3. It states that it "is based on FHIR Version R4" and "defines the minimum constraints on the FHIR resources to create the US Core Profiles", and describes itself as establishing "the 'floor' of standards to promote interoperability and adoption through common implementation". HL7's publication history records fhirVersion 4.0.1 for every US Core release from 3.1.0 onward.

US Core supplies the thing base FHIR withholds. Where R4 leaves the meaning of Must Support to the profile author, US Core writes it down: a Responder SHALL be capable of populating all data elements specified by the US Core Server CapabilityStatement; a Requestor SHALL be capable of processing instances containing those elements without error; a Responder SHALL NOT include an element whose value is absent for an unknown reason; a Requestor SHALL read a missing element as data not present in the Responder's system; a Responder SHOULD send the reason when the reason for absence is known; and a Requestor SHALL be able to process instances asserting missing information. US Core adds a drafting convention on top: all elements with a minimum of 1 are marked Must Support unless they sit under an optional element.

Two further US Core rules change what a test suite has to cover. The guide offers Profile Only Support, where a system implements the profiles and not the interactions, alongside Profile Support with Interaction Support, so a claim of US Core support does not say which was done until someone asks. And it defines a third element category beyond Mandatory and Must Support, called Additional USCDI Requirements, which certifying implementers SHALL interpret as Must Support and everyone else may treat as optional. The set of elements under test is therefore larger for a system seeking certification than for one that is merely US Core conformant.

A conformant server declares itself in the CapabilityStatement. US Core says it SHOULD name the official URL of the US Core Server CapabilityStatement in the instantiates element, SHALL specify the full capability details it claims to implement, and SHALL declare each supported profile by its official URL in CapabilityStatement.rest.resource.supportedProfile. The conformance verbs used throughout that guide are the ones defined in the base R4 Conformance Rules.

In the United States, 45 CFR section 170.215, Application Programming Interface Standards is where the health IT certification programme names the API standards a certified system has to implement, and section 170.315(g)(10) is the criterion that consumes them. Base FHIR R4 makes no reference to that programme anywhere. Which versions that regulation names, and what the criterion measures, is set out in ONC certification testing requirements.

Where do FHIR conformance claims fall apart?

• The CapabilityStatement is hand written once, drifts from the deployed configuration, and no test compares the served document against the endpoints that actually answer.
• kind is left at capability on a deployed instance, so the document fails cpb-15 and describes a product rather than the installation a customer is integrating with.
• A profile is named in the contract with no implementation guide and no version, so validation runs against whichever package the tooling resolved that week.
• Must Support appears in the acceptance criteria and nowhere is it stated which meaning of support was intended, so the criterion cannot be marked pass or fail.
• The suite asserts on HTTP status from $validate, which returns 200 OK for invalid resources, and reports every payload as accepted.
• Warnings and errors from OperationOutcome are collapsed into one number, so a guideline severity flipped on in one environment reads as a regression in another.
• Test data covers the Normative resources and thins out across the FMM 2 and FMM 3 resources that carry the clinical content, which is where a release change lands.
• Slice assertions look for slice names, which are never exchanged, and pass against payloads that carry no discriminating value at all.
• The interface is validated once at go live, and the maturity exposure is never revisited when HL7 publishes a release.

What do we run against FHIR R4?

We start from the served CapabilityStatement rather than from the specification, because that document is the only thing a conformance claim can cover. From it we derive the resource list, the profile canonical URLs, the version pin and the interaction set, then build fixtures and assertions against the frozen baseline and record every run as an OperationOutcome a reader can audit. The artefacts listed below are the output, including the maturity exposure register, which is the part a CTO uses when the next HL7 release is announced.

The interface work itself is described in healthcare interoperability testing. For a product on a release train, where the pinned baseline has to be re-asserted on every build without rerunning the whole suite, regression testing in a regulated environment covers how that evidence is kept current.

Version strings, maturity markers, invariant keys and quoted text on this page were read from the FHIR R4 specification at version 4.0.1 on 2 September 2026.