The pilot ran on forty unresolved cases and the report was favourable: candidates surfaced that the original analysis had set aside, two cases were reopened, the scientists liked it. The pilot lead presented the results, was promoted into a different directorate, and the service went back to the spreadsheet. Two years later the pilot is cited in the strategy document as evidence that the service is innovating.
Nobody decided against adoption, and nobody can now say who would have. Nobody decided for it either, because the questions that adoption would have raised had never been asked. The forty cases are still in the pilot system, under an account that belonged to someone who has left.
What the field knows
The failure is common enough to have a name. Greenhalgh and colleagues (2017) reviewed why technology programmes in health care so often stall after a successful pilot and proposed a framework for the reasons: the condition, the technology, the value proposition, the adopters, the organisation, the wider system and how all of these change over time. Most pilots test the technology and some test the adopters. Almost none test the organisation, which is where adoption is decided, by people who were not in the pilot and have questions the pilot did not answer.
Implementation has outcomes of its own, distinct from clinical ones. Proctor and colleagues (2011) distinguished acceptability, adoption, appropriateness, feasibility, fidelity, cost, penetration and sustainability, and argued that a programme which measures only whether the intervention worked has not measured whether it will still be there in two years. A pilot that reports two reopened cases has reported a clinical outcome and none of the others, and the others are the ones that decide whether there is a third year.
Integration is the usual casualty. Ash, Berg and Coiera (2004) catalogued the unintended consequences of clinical systems that were introduced without regard to how work was actually done, and the pattern recurs in genomics: the tool works in the pilot because the pilot lead moves data into it by hand, and the hand-moving is what the service cannot sustain once the pilot lead has other work. Manolio and colleagues (2013) and Stark and colleagues (2019) both found that the barriers to genomic medicine at scale were organisational before they were scientific.
Where practice breaks
The pilot is measured on enthusiasm. The outcomes are whether the scientists liked the tool and whether anything interesting happened, and both are reported as positive because the people reporting chose to pilot it. What would count as a reason to adopt or to stop is never written down beforehand, so the result cannot fail, and a result that cannot fail cannot persuade anyone who was not already persuaded.
Governance is deferred. The pilot runs on an exception: a data sharing agreement signed for forty cases, a user account shared among the team, no audit trail, a tenant that belongs to nobody. Adoption would require the real arrangement, and the real arrangement is where the information governance questions live, and nobody has started on them. The pilot, by running on an exception, has demonstrated everything except the thing adoption requires.
Integration is deferred. The pilot lead exports from the laboratory system, uploads, and copies results back, and this works for forty cases and not for four hundred. The connectors, the identity integration and the reporting path are described as phase two, and phase two is not funded, because the business case for it rests on a pilot that did not test it.
The exit is never designed. A service cannot commit to a tool it cannot leave, and the pilot never established how the data would come out, in what format, with what history, if the service chose to stop. The absence of an exit is read, correctly, as a risk, and the risk is resolved by doing nothing, which has risks of its own that nobody is asked to sign for, because the cost of a case that stays open appears on no budget line.
How Tulparus handles it
A pilot runs under the production arrangement from the first day. The organisation's tenant is created with its own administrators, its own roles and permissions, its own audit log and its own data residency, so that the governance questions are answered at the start, once, and the pilot is already the service. There is no exception to unwind at adoption, because nothing was done under one, and the information governance review happens before the first case rather than after the fortieth.
The outcomes are agreed before the first case. The pilot plan names what will be counted: cases with a candidate the original analysis set aside, cases reopened, next steps proposed and taken, time from upload to a multidisciplinary decision, and what the scientists did with what was proposed. It also names what would count as a reason to stop, so that stopping is a possible result and the pilot is a test. The counts come from the record, so they are the same numbers the service would see in practice.
Integration is scoped in the plan, with the laboratory's systems named and the connectors agreed, so that the pilot runs on the path adoption would use, or on a path whose difference from it is written down. Identity comes through the organisation's own provider, so that the accounts in the pilot are the accounts the service already manages, and leavers lose access the day they leave. The reporting path is the one the service already uses, and a report produced in the pilot is a report the service could have issued.
The exit is built. A tenant's data is exportable at any time, in standard formats, with the full investigation history, and is erased on request with a manifest. The organisation can leave on the day it chooses, which is the condition under which it can also choose to stay. The pilot above, run this way, would have ended with a decision either way, taken by people who had the numbers, under an arrangement that was already the real one.
What you can do tomorrow
A pilot plan that answers four questions before the first case is the difference between a pilot and a demonstration.
- Write the stopping rule and the adoption rule before the pilot starts, in numbers, and have the clinical director sign them.
- Run the pilot under the governance adoption would need: a real agreement, named accounts, an audit trail. If that is too much work for a pilot, it was never going to be adopted.
- Name the integration the service would need and test at least one connector during the pilot, so that phase two is a known quantity with a cost beside it.
- Ask the vendor to demonstrate a full export of the pilot data, with history, before the pilot ends. Read the export.
Sources
- Greenhalgh T, Wherton J, Papoutsi C, et al. Beyond adoption: a new framework for theorizing and evaluating nonadoption, abandonment, and challenges to the scale-up, spread, and sustainability of health and care technologies. Journal of Medical Internet Research, 2017.
- Proctor E, Silmere H, Raghavan R, et al. Outcomes for implementation research: conceptual distinctions, measurement challenges, and research agenda. Administration and Policy in Mental Health and Mental Health Services Research, 2011.
- Ash JS, Berg M, Coiera E. Some unintended consequences of information technology in health care: the nature of patient care information system-related errors. Journal of the American Medical Informatics Association, 2004.
- Manolio TA, Chisholm RL, Ozenberger B, et al. Implementing genomic medicine in the clinic: the future is here. Genetics in Medicine, 2013.
- Stark Z, Dolman L, Manolio TA, et al. Integrating genomics into healthcare: a global responsibility. American Journal of Human Genetics, 2019.