The information governance lead found it on page fourteen of the terms: de-identified data may be used to develop and improve the provider's services. The laboratory held exomes on eight thousand patients, each consented for diagnostic testing and for nothing else. The vendor, asked, said that the clause was standard, that the data would be de-identified, and that every customer had signed it.
Every customer had. None had asked the patients, and none had asked what improve the service meant when the service was a model.
What the field knows
Genomic data resists de-identification. Gymrek and colleagues (2013) recovered surnames of research participants from their Y-chromosome data and public genealogy databases, and Erlich and Narayanan (2014) catalogued the routes by which a genome, stripped of its name, can be reattached to its owner. A phenotype profile with a rare combination of terms narrows a population to a handful of people; a genome narrows it to one, and a genome with a date of birth and a postcode district is a name. The word de-identified, applied to this data, describes an intention rather than a property, and the law in most jurisdictions is beginning to say so.
Models remember their training data. Carlini and colleagues (2021) showed that large language models can be prompted to reproduce verbatim sequences from the text they were trained on, including personal details that appeared once. A model trained on clinical notes, phenotype descriptions or variant lists can carry fragments of them into its outputs for other users, and the training process offers no mechanism for taking a patient's contribution back out once it is in. Retraining without the data is the only remedy, and no vendor offers it.
Consent and purpose are the governing principles, and the public notices when they are stretched. The GA4GH framework for responsible sharing (Knoppers, 2014) ties every use of genomic data to the consent under which it was collected. Carter, Laurie and Dixon-Woods (2015) examined why a national programme to reuse health records for research failed and concluded that it lost its social licence: the public had agreed to care, and had been enrolled in something else. Price and Cohen (2019) set out how the same tension shapes medical data in the age of large models.
Where practice breaks
Training is buried in terms. The clause sits among the warranties, is drafted to read as housekeeping, and is signed by a procurement officer who has not been told what the data is. The clinical staff who know what the data is never see the contract, and the procurement staff who see the contract have been told the data is anonymous.
Consent for care is read as consent for product development. A patient who agreed to diagnostic testing agreed to have their genome examined for their diagnosis. That the same genome then improves a commercial model for other customers is a different purpose, and nobody asked whether the patient would agree to it, or whether the laboratory's own ethics committee would, because the question was never put in those words. Put in those words, to a room of patients, the answer is frequently no, and the services that have asked have found out.
One organisation's data serves another's cases. A model trained on a laboratory's cohort ranks candidates for a competitor's patients, and the laboratory has given away, for nothing and without knowing, the thing that made its service distinctive. The patients, meanwhile, have become training material for a product that serves people they will never meet.
There is no way out. A patient who withdraws consent, a laboratory that leaves the vendor, a regulator who rules the use unlawful: none of them can remove the contribution from a model that has already been trained on it. The only control that exists is at the point of entry, and the clause gives it away in a sentence.
How Tulparus handles it
By default, no patient data trains any model. The ranking and similarity models are trained on public and licensed reference knowledge, with every source, release and record identifier recorded in the model's lineage, and on nothing from any tenant's cases. The default is the product's and it holds without a clause being negotiated. A tenant that never raises the subject has given nothing away by silence.
An organisation that wishes to contribute does so explicitly, per dataset, under an agreement that names the dataset, the consent it was collected under, the purpose, and the models it may train, and the contribution is recorded in the lineage of every model that used it. A model's lineage lists its training sources and can be read by any tenant, so that a laboratory can see whether any clinical data from anywhere went into the model ranking its cases, and whose. The list is short, and for most models it names no clinical dataset at all.
The language model runs inside the deployment, not at a third-party provider, and it is not trained on tenant data either. What it is sent for a task is the minimum the task needs, and the task, the model release and what was sent are logged on the case, so that a tenant can audit what left the case record and for what. A tenant's data lives in its tenant, is exported on request in standard formats, and is erased on request, with a manifest of what was erased.
For the governance lead above, the terms contain no such clause, the model lineage is a document she can read, and the answer to the question nobody asked the patients is that nothing was done that would have required asking. If the laboratory later chooses to contribute a dataset to a model, it will do so having asked.
What you can do tomorrow
The control is contractual and it is exercised before signature, which means the clinical and governance staff have to be in the room when the terms are read, and the terms have to be read.
- Search every current supplier contract for the words improve, develop, train and de-identified, and list what each clause permits. The exercise takes an afternoon and the results usually surprise the clinical director.
- Ask each supplier for the training sources of any model that ranks or classifies for your patients, in writing. A supplier who cannot answer has answered, and so has one who answers with a description of the de-identification process rather than a list.
- Put a sentence in the laboratory's consent material that says whether patient data is used to develop commercial products, so that the answer is given rather than assumed.
- Make the default in every new contract that no patient data is used for training, and negotiate exceptions as exceptions, with a named dataset and a named purpose.
Sources
- Gymrek M, McGuire AL, Golan D, Halperin E, Erlich Y. Identifying personal genomes by surname inference. Science, 2013.
- Erlich Y, Narayanan A. Routes for breaching and protecting genetic privacy. Nature Reviews Genetics, 2014.
- Carlini N, Tramèr F, Wallace E, et al. Extracting training data from large language models. Proceedings of the 30th USENIX Security Symposium, 2021.
- Knoppers BM. Framework for responsible sharing of genomic and health-related data. HUGO Journal, 2014.
- Carter P, Laurie GT, Dixon-Woods M. The social licence for research: why care.data ran into trouble. Journal of Medical Ethics, 2015.
- Price WN, Cohen IG. Privacy in the age of medical big data. Nature Medicine, 2019.