All articles

Provenance is a clinical safety feature.

Every fact in an interpretation came from somewhere, at some version. Keeping the where and the when is what lets a conclusion be checked when it matters.

A laboratory is audited after a diagnosis it issued is overturned at another centre. The auditors ask a simple question about the original report: for each statement in it, what was the source and which version of that source was consulted? The gene-disease association came from a database, release unknown. The population frequency came from a browser, version unknown. The variant classification came from an archive, downloaded on a date nobody recorded. The in silico predictions came from a tool whose version was in a pipeline configuration that has since been overwritten.

The scientist's reasoning may have been sound on the day. There is no way to show it.

What the field knows

Provenance, in the sense the data management literature uses, is the record of where a piece of information came from and what was done to it. The FAIR principles (Wilkinson et al., 2016) make it a condition of reusable data that each item carry its origin, its version and its licence, so that a consumer can trace a derived result back to its inputs. Clinical genomics consumes an unusual number of such inputs for every conclusion: an ontology, an annotation set, several population databases, a classification archive, a gene-disease resource, a set of predictors, and the literature.

Each of those inputs changes. OMIM (Amberger et al., 2019) adds and revises gene-phenotype relationships continuously. ClinVar (Landrum et al., 2018) issues new releases on a regular schedule in which classifications move. Population databases publish new versions with different sample sets and different allele frequencies, and a frequency that excluded a variant from consideration under one version may not under the next. The laboratory standards (Rehm et al., 2013) ask that the report state the databases and versions used, and the interpretation framework (Richards et al., 2015) assigns evidence weights whose values depend on which version of each source was consulted.

The reanalysis guidance (Deignan et al., 2019) turns the same requirement around: a re-evaluation can only be targeted if the laboratory knows what the first evaluation saw, so that the difference between then and now can be computed. Without recorded versions, every reanalysis is a first analysis.

Provenance is also what makes an error findable. When a source is later found to contain a mistake, a withdrawn association, a mis-mapped variant, a frequency computed from a contaminated sample set, the laboratories that can list the cases that used that source at that version can review them. The laboratories that cannot have to review everything or nothing.

Where practice breaks

Provenance is lost at the point of use. A scientist looks a variant up in a browser and copies a frequency into a worksheet. The browser was showing a version; the worksheet records a number. The pipeline annotates from files on a server; the files have dates in their names, and the annotation output carries values without them. By the time the evidence reaches the report it is a set of facts with no history.

Versions are recorded once, for the pipeline, and assumed to apply to every case. A methods paragraph names the database versions in force when the template was written. Cases analysed a year later inherit the paragraph, and the versions in it are wrong for them, in a document that will be relied on to say what was done.

Conflicts between sources are resolved silently. Two databases disagree about a gene's inheritance pattern, or two population sets give different frequencies; the scientist picks one, and the choice is not recorded, so a later reader sees a single value and cannot tell that a judgement was made. If the chosen source later turns out to have been the wrong one, nothing points at the case.

The consequence for the patient is a conclusion that cannot be defended or corrected. In the audit above, nobody can tell whether the scientist erred, which is a worse position for the laboratory, the scientist and the family than an honest mistake with a traceable cause.

Predictors and pipelines are inputs too, and they are the ones most often left out. An in silico score changes between tool versions, sometimes enough to move a variant across an evidence threshold. A variant caller update changes which variants exist to be interpreted. A report that names the tools without their versions describes the method in the way a recipe without quantities describes a dish.

How Tulparus handles it

No piece of knowledge enters the workbench without its source, its release and its record identifier. The knowledge store loads each external resource as a named release, keeps earlier releases rather than replacing them, and every fact drawn from it carries the release it came from. A frequency on a case is a frequency from a named population set at a named version; a gene-disease association is one from a named resource at a named release; a classification is one submitter's, at one review status, in one archive release.

Each investigation records the full set of releases it was performed against, with the case version and the model versions, before any clinically meaningful analysis runs. The record is immutable: a later investigation on the same case records its own set, and the two can be compared. Conflicts between sources are kept as conflicts, with both values and both provenances on the case, and the scientist's choice between them, when one is made, is recorded as a decision with a reason.

The report is rendered from the case, so the methods it states are the methods that were used for that case, drawn from the investigation record, and the evidence it cites carries its versions. A reader can ask, for any sentence, where it came from, and the workbench can answer. The same holds inside the laboratory: a second reviewer sees each piece of evidence with its source beside it and can disagree with the source as well as with the reading.

When a source is corrected or withdrawn, the workbench can list the cases that used the affected records at the affected releases, because that is what the provenance was kept for. The audit in the opening case would have been an export rather than an archaeology.

What you can do tomorrow

Recording provenance is mostly a matter of writing down what is already on the screen.

  • Record the version of every database consulted, per case, in the case note, and copy it into the report. If the pipeline supplies the values, make it supply the versions beside them.
  • When two sources disagree, write down both and which one you used, and why.
  • Keep the annotation files your pipeline used, named by date, rather than overwriting them on update.
  • Before a reanalysis, list what has changed in each source since the original analysis. If you cannot, treat the recorded versions as the first thing to fix.

Sources

  1. Wilkinson MD, Dumontier M, Aalbersberg IJ, et al. The FAIR Guiding Principles for scientific data management and stewardship. Scientific Data, 2016.
  2. Amberger JS, Bocchini CA, Scott AF, Hamosh A. OMIM.org: leveraging knowledge across phenotype-gene relationships. Nucleic Acids Research, 2019.
  3. Landrum MJ, Lee JM, Benson M, et al. ClinVar: improving access to variant interpretations and supporting evidence. Nucleic Acids Research, 2018.
  4. Rehm HL, Bale SJ, Bayrak-Toydemir P, et al. ACMG clinical laboratory standards for next-generation sequencing. Genetics in Medicine, 2013.
  5. Richards S, Aziz N, Bale S, et al. Standards and guidelines for the interpretation of sequence variants: a joint consensus recommendation of the American College of Medical Genetics and Genomics and the Association for Molecular Pathology. Genetics in Medicine, 2015.
  6. Deignan JL, Chung WK, Kearney HM, Monaghan KG, Rehder CW, Chao EC. Points to consider in the reevaluation and reanalysis of genomic test results: a statement of the American College of Medical Genetics and Genomics (ACMG). Genetics in Medicine, 2019.

Every fact, with its source.See where each sentence came from.

A demo takes 20 minutes, screen shared, on synthetic cases. Bring the question from one of your own.