All articles

Benchmarks you can audit.

Two vendors claim a yield and neither names the set it was measured on. A figure without its evaluation set, sample size and interval is only a sentence.

Two procurement decks, two figures for how often the causal gene appears in the top five. Neither deck named the cases the figure was measured on. Asked, one vendor described an internal set of solved cases that, on inspection, had been drawn from the same cohorts the model was trained on. The other named a public benchmark, and the figure on the slide was higher than the figure in the paper that had introduced the benchmark, with no explanation of what had changed.

The laboratory had no way to tell which claim was closer to the truth, and no way to tell whether either would hold for its own patients, who were older, sicker and differently referred than anyone in either set.

What the field knows

Evaluation leaks. Kapoor and Narayanan (2023) surveyed machine learning across the sciences and found that a large share of reported results were inflated by leakage between training and evaluation data, in forms ranging from duplicated records to features that encode the answer. In variant prioritisation the commonest leak is a solved case that, or whose relatives, appear in the training cohort, and the second is a knowledge base used for evaluation that already contains the gene-disease association the case is meant to test, because the case itself was among the evidence that established it.

The field has built shared benchmarks for exactly this reason. The Critical Assessment of Genome Interpretation (Andreoletti et al., 2019) runs blind challenges in which methods are evaluated on data their developers have not seen, with the results published whatever they show. Phenotype-driven ranking tools report their performance on named sets of solved cases, with the version of the tool and the version of the knowledge, so that another group can reproduce the number (Jacobsen et al., 2022). The reproduction is the point; a figure nobody else can obtain is a claim about the vendor, not about the method.

A figure needs its uncertainty and its scope. Steyerberg and Harrell (2016) set out why a prediction model's performance has to be reported with internal, internal-external and external validation, and why a single number from the development data is the least informative of these. TRIPOD+AI (Collins et al., 2024) asks for the evaluation set's size and provenance, the confidence interval, and the version of the model, and for a statement of the population the figure applies to. A hit rate measured on children with neurodevelopmental disorders says little about adults with cardiomyopathy.

Where practice breaks

The figure travels without its set. A number from a paper, a poster or an internal evaluation is copied onto a slide, and the slide is copied into a tender response, and at each step the sample, the interval and the knowledge version fall away. By the time it reaches a procurement committee the figure is a bare claim, and the committee, which has no way to check it, scores it as if it were a measurement.

The set overlaps the training data. Where a vendor evaluates on its own solved cases, the cases are usually the ones it trained on, or their relatives, or cases whose genes were in the knowledge base because those very cases had been published. The resulting figure is a measurement of memory, and it is reproduced faithfully on every slide.

The figure is read as a property of the patient. A rate measured over a cohort is presented, or understood, as the chance that this patient's gene is in the top five, which it is not and cannot be. A benchmark describes a method on a set, and a patient is one case with its own gaps, its own phenotype and its own untested relatives.

The figure changes with each release and nobody says so. A model update, a knowledge update or a change to the evaluation set moves the number, and the deck is updated without a note of which of the three moved it, so that improvement and leakage are indistinguishable. A committee comparing this year's deck with last year's sees progress and cannot tell what kind.

How Tulparus handles it

Every ranking release ships with a benchmark card, and the card is the only place a performance figure comes from. The card names the evaluation set and how it was built, the number of cases, the hit rates with their confidence intervals, the release of the ranking model and the knowledge snapshot it was evaluated against, and the population the set represents. A figure never appears in the product, in a deck or in a document without the card behind it.

Leakage is checked, and the check is recorded. Before a release is evaluated, the evaluation set is compared against the training data for shared cases, shared families and gene-disease associations that entered the knowledge base through the evaluation cases themselves, and cases that fail the check are excluded with the reason. The card states what was excluded and why, and the number of cases on the card is the number after exclusion.

A release reaches production on the card and on nothing else. A fall in training loss is not a reason to ship; a measured change on the named set, with its interval, against the current production release, is. Where two releases are compared, the comparison is on the same set and the same knowledge snapshot, and the card says so.

Every figure is labelled as benchmark performance and never as a statement about a patient. In the workbench, a candidate shows its rank and the reasons, and a reader who wants to know how the method has performed is sent to the card, with its set and its interval, and a sentence saying that this is the method on that set and not the chance for this case. A laboratory can run the current release against its own solved cases and compare the result with the card, which is the audit the two decks above could not survive.

What you can do tomorrow

The questions below turn a figure back into a measurement, and a vendor who cannot answer them has given you the answer.

  • For every performance figure in a tender response, ask for the evaluation set, its size, the interval, the model version and the knowledge version. Decline to score the figure until you have them.
  • Ask how the evaluation set was checked against the training data for shared cases and families, and what was excluded.
  • Run the tool on twenty of your own solved cases, chosen by someone who has not seen the vendor's set, and compare the result with the claim.
  • Strike any sentence in your own service's documents that reports a benchmark rate as a probability for a patient. The two are different things, and a clinician will read the second.

Sources

  1. Kapoor S, Narayanan A. Leakage and the reproducibility crisis in machine-learning-based science. Patterns, 2023.
  2. Andreoletti G, Pal LR, Moult J, Brenner SE. Reports from the fifth edition of CAGI: the Critical Assessment of Genome Interpretation. Human Mutation, 2019.
  3. Jacobsen JOB, Kelly C, Cipriani V, et al. Phenotype-driven approaches to enhance variant prioritization and diagnosis of rare disease. Human Mutation, 2022.
  4. Steyerberg EW, Harrell FE. Prediction models need appropriate internal, internal-external, and external validation. Journal of Clinical Epidemiology, 2016.
  5. Collins GS, Moons KGM, Dhiman P, et al. TRIPOD+AI statement: updated guidance for reporting clinical prediction models that use regression or machine learning methods. BMJ, 2024.

Two decks, two figures.See the card behind the number.

A demo takes 20 minutes, screen shared, on synthetic cases. Bring the question from one of your own.