Numbers that look like chances
Software that ranks genes usually shows a number next to each one. It might be 0.84, or 92, or three stars. A reader sees a number between zero and one and reads it as a chance. Eighty-four percent that this is the gene.
In almost every case that is not what the number is. It is a score that puts candidates in an order, built from how well each one fits the case. It tells you which gene to look at before which. It does not tell you how likely any of them is to be the answer, and it does not add up to one across the list.
A small example
Imagine ranking five runners by how fast they trained this month. The order is useful. The fastest trainer is a sensible first guess for who wins. Now ask what the chance is that the first runner wins on Saturday. The ranking does not say. It does not know about the course, the weather or who got injured on Friday, and a runner who trained very fast could still be one of ten with a similar time.
A gene ranking has the same shape. The top gene fits the case best by the measures used. Whether it is the patient’s answer depends on things the score never saw, and in cases that have already been through testing, the answer is often not near the top of any list built from known links.
What we do instead
In the product, a score or a rank is never shown as the chance that a gene is this patient’s diagnosis. A rank is shown with its reasons: what in the record supports it, what goes against it, and what is still unknown. The reasons are the useful part, because they are what a person can weigh against everything they know about the patient and the family.
Where we do show a benchmark figure, it is labelled as a measured performance on a stated evaluation set, with how many cases it covered, an interval around the figure, and which release and knowledge snapshot produced it. It is never the chance for the patient in front of you. It describes how a version of the system did on a set of cases that had answers, and nothing more.
A word on benchmarks
Benchmark figures are hard to compare and easy to over-read. A few general points are worth knowing whoever is reporting them.
- Published solved cases are not typical open cases. They are the ones that did get solved, and often the ones with the clearest picture.
- A system built on the literature may have seen the very cases it is then tested on, in the papers that reported them.
- A hit rate depends on how far down the list you are willing to look. “In the top ten” and “first” are different claims.
- A figure on a set of a few dozen cases comes with a wide interval, and the interval matters as much as the figure.
For these reasons we do not put performance figures on this site.
What a team should take from a list
Use the order to decide where to look first, and use the reasons to decide whether to keep looking. When a rank and a clinician’s judgement disagree, the clinician has information the ranking does not. That is the design, and it is the reason the final step in the product belongs to a person.
Why the number is still shown in some form
A fair question is why a score exists at all if it is easy to misread. The answer is that order is useful and a list has to be in some order. A clinician with limited time wants to start somewhere, and a sorted list with reasons beats an unsorted pile.
What we try to avoid is the number doing the persuading. Next to a score the product shows what the score is for: it orders hypotheses within this case, and it is not a probability. The wording is deliberate and it stays on the screen, because a number with a plausible shape will be read as a chance whatever the footnote says.
Questions worth asking of any ranking tool
- What do the numbers mean, in one sentence, and who checked that sentence?
- What cases was it tested on, and how were they chosen?
- Can I see why a candidate is at the top, in terms I can verify?
- What does it do when a feature was never examined?
- Who can change a result, and is it recorded?
We would like to be asked these questions, and to be able to answer them.