All articles

Decision support, and where the decision stays.

A tool that proposes and a person who decides works only if the record shows which was which. Most systems blur the line, and always in one direction.

The procurement questionnaire asked, in one box, whether the system makes the diagnosis. The clinical director's worry, in the meeting that followed, was the opposite of the vendor's pitch: a tool good enough to be trusted would be trusted, the scientist's role would shrink to approving what appeared on the screen, and when something went wrong there would be no record of who had decided what.

The senior scientist put it more plainly. She did not want to be a signature at the bottom of a page the software wrote, and she wanted to know, before the contract, how the system would stop that from happening by degrees.

What the field knows

The hazard has a name and a literature. Bainbridge (1983) described the ironies of automation: a system that does most of the work leaves the person with the hardest part, catching its rare failures, while removing the practice that would keep them able to. Goddard, Roudsari and Wyatt (2012) reviewed automation bias in clinical decision support and found that people given a system's recommendation accept it more often than its accuracy warrants, and that the effect grows with workload and with the confidence of the display. A busy laboratory with a confident tool is the condition under which the effect is largest.

Regulators have drawn a line between support and decision. The United States Food and Drug Administration's guidance on clinical decision support software (2022) distinguishes software that presents information and the basis for a recommendation, so that the clinician can independently review it, from software whose output the clinician cannot reasonably evaluate, and treats the second as a device. The distinction turns on whether a person can see the reasons and could disagree, and it is a design property before it is a legal one.

Evaluation frameworks have followed. DECIDE-AI (Vasey et al., 2022) sets out what an early clinical evaluation of a decision support system should report, including how the human and the system interact, how often the person overrides the system and why, and what happens to the person's performance over time. Topol (2019) made the broader case that the value of such systems in medicine lies in what they return to the clinician, time and attention, and that a system which takes the decision has taken the wrong thing. The arrangement that works is one in which the system does the part a person does badly, the searching and the bookkeeping, and the person does the part the system cannot be held to account for.

Where practice breaks

Tools are designed as oracles. The screen shows an answer, with a number, and the controls are accept and reject. The person's contribution is reduced to a click, and a click is what the record holds. The design has decided, before any scientist sits down, how much thinking the screen expects. The design has decided, before any scientist sits down, how much thinking the screen expects. Whether the scientist agreed after an hour's reasoning or after a glance is indistinguishable afterwards, to the service, to an auditor, and to the scientist herself a year later.

The proposal and the decision are stored as one thing. When the tool's candidate becomes the report's finding, the record does not say that the tool proposed it and the person accepted it; it says the finding. A service cannot later ask how often its scientists diverged from the tool, because divergence was never a recorded event. The one number that would show whether the arrangement is working is the one number the system cannot produce.

The tool's text enters the report. A generated summary is pasted into the clinical document, edited lightly or not at all, and the clinician who reads it cannot tell which sentences a person wrote. When a sentence is wrong, nobody can say who wrote it, and the correction, when it comes, is to the report and never to the process that produced the sentence.

Accountability is assumed to sit with the signatory, and the signatory was given no way to exercise it. A person asked to sign for a conclusion whose reasoning they could not see has been given responsibility without the means, which is the arrangement the regulators and the human factors literature both warn against.

How Tulparus handles it

Tulparus is a decision support and investigation system, and the design keeps the decision where it belongs. The workbench computes, retrieves, compares and proposes: it ranks candidates and shows the reasons, lists the gaps in the evidence, suggests the next most informative step, and retrieves the literature and the knowledge for a hypothesis. It makes no clinical statement of its own. Every hypothesis is written by a named person, every classification is a person's with the criteria they applied, and every report is approved by a person whose name and date are on it.

Proposals and decisions are separate records. What the system proposed is stored as a proposal, with its version and its reasons; what the person decided is stored as a decision, with the person and the time. A service can report, for any period, how often its scientists accepted, modified or rejected what was proposed, which is the measure DECIDE-AI asks for and the measure a clinical director needs to know whether the arrangement is working.

No text reaches a report without a person. The report is rendered from the structured investigation record, and the narrative parts of it are written or explicitly adopted by the signatory, with the generated draft kept separately so that a reader can see what was proposed and what was signed. A reanalysis that finds something new offers a re-draft; it never drafts on its own.

For the clinical director above, the answer to the procurement question is that the system does not make the diagnosis, and that the record will show, for every case, what it proposed and what her staff decided, in a form she can audit without asking the vendor.

What you can do tomorrow

The questions to ask of any decision support tool in use or under procurement follow from the literature, and the answers are usually short. A vendor who has thought about the problem will have them ready; one who has not will describe the accuracy of the model instead.

  • Ask whether the record distinguishes what the tool proposed from what the person decided. If it does not, the service cannot measure its own override rate, and should.
  • Ask how a scientist disagrees with the tool, what the screen offers for it, and where the disagreement is stored.
  • Trace one report back to its sources and mark which sentences a person wrote. If you cannot, the tool is writing your reports.
  • Write the service's own statement of where the decision sits, and show it to the vendor before the contract.

Sources

  1. Bainbridge L. Ironies of automation. Automatica, 1983.
  2. Goddard K, Roudsari A, Wyatt JC. Automation bias: a systematic review of frequency, effect mediators, and mitigators. Journal of the American Medical Informatics Association, 2012.
  3. U.S. Food and Drug Administration. Clinical decision support software: guidance for industry and Food and Drug Administration staff, 2022.
  4. Vasey B, Nagendran M, Campbell B, et al. Reporting guideline for the early-stage clinical evaluation of decision support systems driven by artificial intelligence: DECIDE-AI. Nature Medicine, 2022.
  5. Topol EJ. High-performance medicine: the convergence of human and artificial intelligence. Nature Medicine, 2019.

The system proposes.See who decided, on every case.

A demo takes 20 minutes, screen shared, on synthetic cases. Bring the question from one of your own.