All articles

The reranker should explain itself.

A learned model moved a gene from fourteenth to second. If the only answer to why is that it weighed many features, it has no place in a clinical record.

The candidate list came back and a gene the scientist had never heard of sat in second place, up from fourteenth on the plain phenotype ranking. She asked what had moved it. The documentation said the reranker weighed several hundred features learned from solved cases. The support engineer said the model had found a pattern. The scientist wrote the gene into the report as a candidate, because it was second, and could not have told the clinician why.

Six months later a new model version moved it back to eleventh. Nobody told her that either. The report, by then, was in the notes, with the gene in it.

What the field knows

Learned ranking improves on hand-built scoring in most published evaluations of variant prioritisation. Models that combine phenotype similarity, variant annotations, gene-level evidence and, in some systems, the literature (Birgmeier et al., 2020) place the causal gene higher on average than any single component. The gain is real and it is the reason such components are built. A scientist with forty candidates and an afternoon benefits from a better order, and the benefit compounds across a service.

The gain comes with a cost that the clinical machine learning literature has spent a decade describing. Rudin (2019) argued that for high-stakes decisions an interpretable model, one whose reasoning a person can follow, should be preferred over a black box with a post hoc explanation, and that the accuracy gap between them is usually smaller than assumed. Ghassemi, Oakden-Rayner and Beam (2021) went further on the explanations themselves: feature attribution methods produce plausible stories that may bear little relation to what the model computed, and a clinician given such a story is reassured without being informed.

Reporting standards have followed. TRIPOD+AI (Collins et al., 2024) asks that a prediction model used in health care describe its training data, its features, its evaluation set, its performance with uncertainty, and its version, and that changes to the model be reported as changes. A ranking model that affects which gene a scientist examines first is a prediction model by any reasonable reading, and most deployed rerankers would fail the checklist at the first item. The standard exists because the field has watched models enter practice with none of these things stated, and has watched what followed.

Where practice breaks

The rank moves and the reason is unavailable. A scientist who cannot say why a candidate rose cannot defend it in a multidisciplinary meeting, cannot write it into a report in a way a clinician can use, and cannot tell whether the rise reflects the patient or an artefact of the training set. The candidate is in the report because of its position, and the position is unexplained, which is the situation the rulebook was written to prevent.

The features include things the scientist would reject if shown them. A model trained on solved cases learns that well-studied genes are solved more often, because they are, and promotes them. The scientist, asked directly, would say that a gene's publication count is no evidence about this patient. The model is never asked directly, and so it is never told no.

The model changes silently. A version update, pushed to improve average performance, changes individual rankings in ways nobody reviews, and a case analysed in March and reanalysed in September gets two different lists with no record of what changed between them. The report cites neither version.

Post hoc explanations are offered in place of design. A feature importance chart is shown beside the rank, and it differs each time it is computed, and it is read as the reason when it is a guess about the reason. The chart satisfies the question without answering it, which is worse than no chart.

How Tulparus handles it

Every input to the reranker is a named, deterministic quantity computed before the model runs and shown on the candidate: the phenotype similarity from each channel, the variant's population frequency and predicted effect, whether the inheritance in the family is consistent with the gene's known pattern, the gene-disease validity classification, the similar-case signal, and whether the gene has disease features that were never assessed in this patient. The model reorders candidates on these quantities and on nothing else. The list of inputs is short enough to read on one screen, and it is the same list for every case. There are no hidden features, and a feature that a scientist would reject as evidence about a patient, such as how often the gene has been published, is not an input.

The baseline rank is always shown beside the reranked one. A candidate that the model moved displays how far, and the quantities on which it differs from the candidates it passed, so that the scientist reads the move as a comparison she can check rather than a verdict she must accept. A gene that rose on a strong similar-case match and fell on inheritance inconsistency shows both.

The model is a release, pinned on the case. The case records which reranker release produced each list, and a reanalysis with a newer release shows the two lists side by side with the changes named. A release reaches production only after it has been measured against a named benchmark set with a confidence interval, and never because a training loss fell; the benchmark, its size and its release are shown wherever a performance figure appears, labelled as benchmark performance and never as a statement about this patient.

For the scientist above, the move from fourteenth to second would have shown as a strong similar-case match in the organisation's own records, with the matched case identified by its identifier, and the scientist would have opened it, read it, and either written the candidate up with that reason or dismissed it with one. When the model release changed six months later, the case would have shown the new list beside the old, with the candidate's fall and the quantity that caused it.

What you can do tomorrow

The questions below can be asked of any ranking tool a service already uses, and the answers decide whether its output belongs in a clinical record.

  • Ask the vendor for the full list of input features and the version of the model that produced each ranking. If either is unavailable, record the rank as a tool output and never as evidence, and say so in the report.
  • For each case, keep the plain phenotype ranking beside the learned one and write down, for the top three, what moved them.
  • Record the model version in every report that used it, and treat a version change as a knowledge change that may warrant reanalysis.
  • When a feature importance display is offered, run it twice. If it differs, stop showing it to clinicians.

Sources

  1. Birgmeier J, Haeussler M, Deisseroth CA, et al. AMELIE speeds Mendelian diagnosis by matching patient phenotype and genotype to primary literature. Science Translational Medicine, 2020.
  2. Rudin C. Stop explaining black box machine learning models for high stakes decisions and use interpretable models instead. Nature Machine Intelligence, 2019.
  3. Ghassemi M, Oakden-Rayner L, Beam AL. The false hope of current approaches to explainable artificial intelligence in health care. The Lancet Digital Health, 2021.
  4. Collins GS, Moons KGM, Dhiman P, et al. TRIPOD+AI statement: updated guidance for reporting clinical prediction models that use regression or machine learning methods. BMJ, 2024.
  5. Smedley D, Jacobsen JOB, Jäger M, et al. Next-generation diagnostics and disease-gene discovery with the Exomiser. Nature Protocols, 2015.

Fourteenth to second.See every reason the rank moved.

A demo takes 20 minutes, screen shared, on synthetic cases. Bring the question from one of your own.