Research
What we test,and how we report it.
We have not published a paper yet. This page lists what we plan to write up, how we evaluate the models, and what we will not claim from the results.
Publications
Not published yet.
Each write-up below is in preparation. We will link it here when it exists, and not before.
In preparationUnknown is not absent: phenotype matching with four statesWhat happens to a match when a feature that was never examined is counted as absent, and what changes when it is not.
In preparationCandidate new genes, kept on their own listWhy genes with no established disease link are ranked apart from the established ones, and how that list is evaluated on cases reported later.
In preparationA rank is not a probabilityExplaining a rank with its sources instead of a score, and what a benchmark hit rate does and does not say about one patient.
In preparationKnowledge releases and the second lookRe-checking only the cases a change in the knowledge could touch.
Evaluation
How we test a model.
The rules for a result are fixed before the run, not after.
Frozen evaluation setsModels are compared on fixed sets of published cases, with the knowledge limited to what was available when each case was reported.
Written down firstThe protocol and its checksum are recorded before a run starts, so a result cannot move the goalposts.
Size and intervalA hit rate is reported with the evaluation set, the number of cases, a 95% interval, the model release and the knowledge snapshot.
Benchmark, not a patientA benchmark hit rate describes a release on a set of cases. It is never the chance that a gene or disease is the answer for the patient in front of you.
Loss is not a reasonA new model replaces the current one only if it does better on the evaluation set. A lower training loss is not enough.
Principles
What the models are held to.
Deterministic firstCalculate what can be calculated. Use a model where judgement is needed, and say so.
Evidence is immutable and sourcedEvery claim keeps its source, release and record id.
Missing is not absentPresent, absent, unknown and not tested are four different values.
Versioned and reproducibleAny result can be re-run on the release that produced it.
People signNo gate passes without a person.