All articles

One ranking channel is one point of failure.

Every prioritisation method has blind spots set by its inputs. A case ranked by one method inherits them all. Several, fused and shown apart, do not.

A laboratory ranks every exome with one phenotype-driven tool and reviews the top twenty genes. For a boy with a progressive ataxia, mild cognitive difficulty and a brother with similar but milder signs, the tool's top twenty are dominated by genes for early-onset ataxias with features he does not have. The gene that explains him, with an atypical presentation described in two case reports, is at position sixty-three, because the disease's annotated phenotype in the ontology was written from the classical form.

A literature search on his phenotype terms would have put the gene in the first ten. Nobody ran one, because the ranking was the ranking.

What the field knows

Candidate ranking methods differ in what they read. Exomiser (Smedley et al., 2015) scores a gene by the semantic similarity between the patient's HPO terms and the terms annotated to the gene's diseases, and to model organism phenotypes for genes with no human disease yet. LIRICAL (Robinson et al., 2020) computes likelihood ratios per feature against each disease. Phrank (Jagadeesh et al., 2019) uses information content across the ontology to score term sets. AMELIE (Birgmeier et al., 2020) reads the primary literature and ranks genes by how well published case descriptions match the patient. Each method draws on a different source, and each is only as complete as that source.

Benchmarks make the complementarity visible. Yuan and colleagues (2022) evaluated phenotype-driven prioritisation methods on the same sets of solved cases and found that no single method led on every case type; the methods disagreed on individual cases far more than their average performance suggested, and the gene one method placed low another placed high. The pattern is familiar from information retrieval, where combining rankings from different systems with a simple rule such as reciprocal rank fusion (Cormack, Clarke and Buettcher, 2009) is consistently stronger than the best single system, because the systems fail on different queries.

The failure modes are predictable from the inputs. An ontology-based method fails where the disease annotation is incomplete or written from the classical presentation. A literature-based method fails for a gene whose case reports use different words, or have not yet been written. A population-constraint signal fails for recessive genes, which tolerate heterozygous loss. A variant-pathogenicity predictor fails for the variant classes it was not trained on. None of these is a flaw to be fixed. They are the shape of the evidence each method reads.

Comparing patients with patients is a further channel with its own shape. Matching a phenotype against other solved cases rather than against disease annotations finds the atypical presentation that the annotation has not caught up with, because another clinic has already seen it. It fails for the condition nobody has solved yet, which is where the annotation-based and literature-based channels, reading model organism data and new case reports, still have something to say.

Where practice breaks

Most laboratories run one ranking method, because one was validated, licensed, integrated and taught. The method's blind spots become the laboratory's blind spots, and because the same method is used at reanalysis, the same cases are missed in the same way every time. A case that the method ranks badly is ranked badly for years.

The review threshold compounds it. A scientist reviews the top twenty or thirty genes; what sits below is not seen. A gene at position sixty-three under one method and position eight under another is reviewed or not depending on which method the laboratory bought. The decision was made at procurement, by people who were not thinking about this boy.

Validation makes the lock-in respectable. A method validated on a laboratory's solved cases is validated on the cases it was already good at, since those are the ones that were solved. Its blind spots are, by construction, under-represented in the validation set, and the laboratory concludes from a strong result that the method is sufficient.

When a second method is available it is often run ad hoc, by the scientist who happens to know it, and its result lives in a spreadsheet beside the official one. The two lists are reconciled by eye, the reasoning is not recorded, and the next scientist does it differently or not at all.

There is also a quieter failure of reading. A single score, presented as a number, invites the reader to treat it as a probability that the gene is the answer. It is a similarity under one method's assumptions. Two methods that disagree make that plain; one method that speaks alone does not.

How Tulparus handles it

The workbench generates candidates through several channels and never through one. Phenotype similarity against disease annotations, literature matching on the patient's terms, variant-level evidence, inheritance fit from the pedigree, and similarity to other solved cases each produce their own ranked list, from their own sources, with their own knowledge release recorded. The lists are fused into one order for review, and the fusion is shown, so that for every gene a reader sees which channels raised it and which ignored it.

A gene that one channel places high and the others do not is not buried by the average. The fused order carries a mark for disagreement, and the review view lets the scientist sort by any single channel or by the gap between them. For the boy above, the literature channel would have placed the gene in its first ten while the ontology channel placed it low, and that disagreement is the kind of entry the workbench puts in front of a person.

No channel's score is shown as a chance that the gene is the diagnosis. Each is labelled with what it measures: similarity to annotations, match to published cases, constraint, segregation. The fused rank is an order of attention, and the page says so where the number is.

Because every channel's output is stored with the case, reanalysis can be a channel at a time. When the literature index is updated, the literature channel is re-run and its changes listed; when the ontology annotations change, the phenotype channel is. A case is reopened by the channel that changed, with the reason.

What you can do tomorrow

Two methods are better than one even when the second is run by hand.

  • Run a second, differently built prioritisation method on your next ten negative cases and compare the top twenty lists. The genes on one list and not the other are where to look.
  • Record which method ranked each reviewed gene, and at what position, in the case record.
  • For a negative case, note the highest-ranked genes that were not reviewed because of the threshold. They are the first candidates at reanalysis.
  • Stop presenting a single score as the chance that a gene is the answer, in reports and in conversation. Call it what it is, an order of attention under one method.

Sources

  1. Smedley D, Jacobsen JOB, Jäger M, et al. Next-generation diagnostics and disease-gene discovery with the Exomiser. Nature Protocols, 2015.
  2. Robinson PN, Ravanmehr V, Jacobsen JOB, et al. Interpretable clinical genomics with a likelihood ratio paradigm. American Journal of Human Genetics, 2020.
  3. Jagadeesh KA, Birgmeier J, Guturu H, et al. Phrank measures phenotype sets similarity to greatly improve Mendelian diagnostic disease prioritization. Genetics in Medicine, 2019.
  4. Birgmeier J, Haeussler M, Deisseroth CA, et al. AMELIE speeds Mendelian diagnosis by matching patient phenotype and genotype to primary literature. Science Translational Medicine, 2020.
  5. Yuan X, Wang J, Dai B, et al. Evaluation of phenotype-driven gene prioritization methods for Mendelian diseases. Briefings in Bioinformatics, 2022.
  6. Cormack GV, Clarke CLA, Buettcher S. Reciprocal rank fusion outperforms Condorcet and individual rank learning methods. Proceedings of SIGIR, 2009.

Five channels, one list.See where they disagree.

A demo takes 20 minutes, screen shared, on synthetic cases. Bring the question from one of your own.