FHIR conformance testing tools and what each one proves

Which FHIR conformance testing tool proves what?

Each tool proves one layer, and R4 states that no tool proves conformance. The specification publishes a capability matrix of six methods: two schema routes, one driven by XML Schema alone and one driven by XML Schema with Schematron, then JSON Schema, ShEx, the Java validator and the validation operation. HL7's own verdict covers all six at once. They are incomplete, they reach only the computable part of conformance, and the rules an implementation guide writes in narrative sit outside every one of them. The page carrying that matrix is Informative and imposes no requirement itself.

Two further statements decide how a run may be used in a supplier conversation. Checking resource content statically does not prove conformance to the specification, in R4's own words. And the arbiter R4 nominates when methods disagree is a person reading the resources against the applicable guides and the base specification. R4 endorses no conformance testing service either: for testing past resource validation it points the reader at a page on a different host, outside the specification. A tool run is evidence about instances, while the conformance claim itself is a statement about one CapabilityStatement, which FHIR R4 conformance testing requirements sets out in full.

Reach differs sharply between the six, and R4 says where.

What R4 states about the reach of each validation method in its matrix:

The six validation methods in the FHIR R4 matrix, each set against what the R4 specification itself states about its reach.

Method	What R4 states about it
XML Schema	Part of the matrix, and named with Schematron as the least capable route, because neither is connected to a terminology server
XML Schema with Schematron	Entry point fhir-invariants.sch, which "includes all the schematrons"; XSLT2 is needed to run them
JSON Schema	Part of the matrix, and covered by the statement that every listed method is incomplete
ShEx	Part of the matrix, and covered by the same statement
The FHIR Validator	A Java jar published with the specification, used in its own publication process to validate the published examples; it "requires an underlying terminology server"
The $validate operation	Normative in R4 from version 4.0.1; returns exactly one OperationOutcome; servers supporting it will usually only validate against profiles they already hold

Terminology is where a difference between methods becomes a difference in what passes. Under a binding of strength required, the code in the element has to come from the value set the profile names, so checking the element means resolving that value set somewhere. A method with no terminology server behind it cannot make the check at all, and R4 ties the quality of the Java validator both to the definitions loaded into it and to whether its terminology server covers the terminologies in play.

That dependency has a licensing edge. The specification is published under Creative Commons "No Rights Reserved", and its licence page states that accepting those terms "does not grant any rights with respect to Third Party IP", listing SNOMED CT, DICOM and LOINC among the artefacts that may need a licence of their own. A terminology server carries the code systems its operator is licensed to carry. The verdict a run returns on a SNOMED CT binding therefore depends on a contract as much as on a configuration.

Terminology also has operations of its own, separate from resource validation. ValueSet/$validate-code became Normative at version 4.0.1, and it answers whether a coded value sits in the set a value set allows, returning a true or false result, an error message and the recommended display for the code. Five normative terminology operations sit in the ANSI Terminology and Conformance package in total: CodeSystem lookup, CodeSystem subsumes, CodeSystem validate-code, ValueSet expand and ValueSet validate-code. An argument between two integration teams about one code is closed by calling one of those against a named server, and two resource validation reports read side by side will not close it.

How do you set up a validator run you can defend?

Pin every input before the first run, then prove the run is able to fail. The FHIR Validator takes the specification version through a -version flag, a profile through a -profile flag carrying a canonical URL, and an implementation guide through an -ig flag when the profile is published inside a guide. A resource can be handed to it as an http address in place of a local file. Each of those inputs changes the verdict, so each one belongs in the record of the run.

Steps 7 and 8 are the pair teams skip. A validator that resolved no profile, or that fell back to the base resource definition without saying so, reports a clean instance and looks identical to a validator that checked everything you asked for. A planted defect is the cheap way to tell the two apart. It has to violate a constraint the profile adds, since a defect that violates a base rule is caught by a validator with no package loaded at all.

One caution about the jar, taken from where R4 sends the reader to fetch it. The download link on the validation page points at the continuous integration build instead of a versioned R4 artefact, and the same page names a single default terminology server run by a third party. Both statements sit in a page whose build stamp is dated November 2019. Check the download address and the default server on the day the pipeline is built, and record which binary produced the evidence.

What does a tool run produce?

One OperationOutcome, carrying issues at four severities that rank the run as much as the instance. The issue-severity code system is Normative from 4.0.0 and defines exactly four codes. A fatal issue stops the checking, so nothing beyond it was examined. An error is important enough to fail the action. A warning leaves the resource valid and the action performed less well than intended. An information issue bears on success not at all.

A short issue list is not a good result on its own. After a fatal, the run ended early and the rest of the instance was never read. A pass mark cannot encode where a run stopped, which is the argument for filing the outcome document itself.

Constraint severity is a second and separate scale, and it is what lets two honest runs disagree. R4 grades constraints at three levels. An Error is a rule every resource has to meet. A Warning flags a possible problem and leaves the resource valid. A Guideline is a Warning tagged with the elementdefinition-bestpractice extension, and it counts as an error only where the deployment has asked for best practice enforcement. A report that does not state whether that enforcement was on cannot be compared with the report from the next release.

One of the eight categories a validation run is meant to cover is defined so that no package can carry it. R4 calls that category business rules and describes it as rules made outside the specification, offering duplicate detection, reference resolution and authorisation of the requesting user as its examples. Those assertions belong to the test suite an EHR or EMR platform runs against its own interfaces, and none of them belongs in a line item that reads "validator run, passed".

R4 publishes an executable test artefact of its own, and its status governs how much weight it will carry. TestScript is at Trial Use. TestReport, the record a run leaves behind, sits at maturity level 0. The page documenting the execution model of both is itself at standards status Draft. What that buys in exchange is an artefact the other side of an integration can read and run without owning your framework, with the fixtures, the profile references and the assertions declared inside the artefact. Where that trade lands for a product shipping every fortnight is part of the wider question in test automation for healthcare products.

One detail of that model quietly disables checks. Conditional assertions have no native support, and the workaround R4 documents is to set warningOnly to true on the assertions that need them. An assertion in that state reports and never fails the run, so a suite can hold a whole class of check that no release can trip.

Fixtures are the last input, and they decide who else has to be in the room. R4 treats every fixture a script declares as required for that script to execute, so the data is not optional scaffolding. A fixture cut from a live system carries whatever the live system holds, and so does a resource posted to a hosted $validate endpoint, which R4 notes some public test servers offer. Settle what the fixtures may contain before either happens: what we sign before PHI is involved, and what our testing runs on answers that from this side.

Fixtures borrowed from the specification carry a caveat R4 states about itself. If an example disagrees with the specification, the specification is correct and the example is not, and the same holds for the reference implementations. The Version Management Policy adds that changes to examples are never treated as substantive. A published example is a starting shape for a fixture, and a run measured against it is measured against nothing binding.

Can a validator run satisfy ONC certification testing?

No. A tool sits inside the programme's own conformance method as one component beside documentation and a live demonstration, and the testing itself is conducted by a laboratory ONC has authorised. Two organisations held that authorisation on 2 September 2026, Drummond Group and SLI Compliance. Your own runs shorten the list of findings the laboratory raises, and they decide nothing.

What a validator run is genuinely good for here is the package pin, and the API criterion at section 170.315(g)(10) makes that pin awkward. The criterion measures a product against the US Core Server CapabilityStatement, and against every element the adopted standards and implementation specifications mark mandatory or must support. The base standard behind it is FHIR Release 4.0.1. The implementation guide version is the open question: the regulation carries US Core STU 6.1.0, while the 2026 cycle of the Standards Version Advancement Process approved STU 9.0.0 for voluntary use, and STU 3.1.1 lost its adoption on 1 January 2026. Three package versions are therefore in circulation for one criterion.

A validator does not know which of the three applies to you. Load the wrong package and the report fills with element-level findings against a profile set nobody is measuring the product on, while the findings that matter are absent. So the SVAP decision has to be taken before the tool chain is configured, and it has to be written into the test plan next to the package file. What the criterion asks for, who performs the testing and how the version pins interact are set out in how ONC certification testing is conducted, and by whom.

Where does a FHIR tool run mislead you?

• The package version is never pinned, so the validator resolves whichever copy of the guide the tooling had cached, and two engineers produce two reports from one instance.
• No planted defect is kept, so a run that loaded no profile at all reads as a clean product.
• The terminology server behind the run holds no licence for SNOMED CT, and every required binding against it comes back as something other than a failure.
• The jar is pulled from the continuous integration build on the morning of the run, so the evidence names no binary anyone can fetch again.
• The report is filed as a pass mark and the outcome document is thrown away, which leaves a later disagreement about what was checked with nothing to settle it.
• Conditional assertions across a TestScript suite are all set to warningOnly, so checks exist that no release can trip.
• Published examples are used as fixtures and then treated as the expected shape, against R4's own statement that its examples do not bind.
• Duplicate detection, reference resolution and authorisation checks are left to the validator, which never covered them.
• The run happens once before a release and never again after a package update, so a profile change reaches production with no report against it.

The same reading problem exists one standard back, in a different vocabulary. On the v2 side the object a validator validates against is an XML conformance profile, and HL7's conformance methodology holds that any content the message profile does not explicitly permit is a conformance violation, while a live receiver may choose to ignore that violation and process the message anyway. A tool that reports both without distinguishing them will fill a defect queue with interfaces that work. Estates running both protocols are answered in HL7 v2 conformance testing requirements.

Tool output starts being useful to a buyer at the point where every run names its package version, its version flag, its terminology server and its fixtures, and where the outcome document survives the release that produced it. Building that record around an integration estate already in production is the work described in interoperability testing for healthcare interfaces.

Quoted text, maturity markers, operation contracts and regulation citations here were taken from version 4.0.1 of the specification, from healthit.gov, and from the text of 45 CFR part 170 in force on 31 August 2026. All three were read on 2 September 2026.