DeepMind’s Latest A.I. Health Breakthrough Has Some Problems

The Google machine learning company trumpeted its success in predicting a deadly kidney condition, but its results raise questions around data rights and patient diversity

Demis Hassabis, software developer and co-founder of DeepMind Technologies, speaks during the DLD conference in Munich, Germany, 16 January 2017. Photo: picture alliance/Getty Images

Google-affiliated artificial intelligence firm DeepMind has been pushing into the healthcare sector for some time. Last week the London-based company synchronized the release of a set of new research articles — one with the U.S. Department of Veterans Affairs, and three with a North London hospital trust known as the Royal Free.

In one paper, published in the journal Nature, with co-authors from Veterans Affairs and University College London, DeepMind claimed its biggest healthcare breakthrough to date: that artificial intelligence (A.I.) can predict acute kidney injury (AKI) up to two days before it happens.

AKI — which occurs when the kidneys suddenly stop functioning, leading to a dangerous buildup of toxins in the bloodstream — is alarmingly common among hospital patients in serious care, and contributes to hundreds of thousands of deaths in the United States each year. DeepMind’s bet is that if it can successfully predict which patients are likely to develop AKI well in advance, then doctors could stop or reverse its progression much more easily, saving lives along the way.

Beyond the headlines and the hope in the DeepMind papers, however, are three sober facts.

First, nothing has actually been predicted — and certainly not before it happens. Rather, what has happened is that DeepMind has taken a windfall dataset of historic incidents of kidney injury in American veterans, plus around 9,000 data-points for each person in the set, and has used a neural network to figure out a pattern between the two.

Second, that predictive pattern only works some of the time. The accuracy rate is 55.8% overall, with a much lower rate the earlier the prediction is made, and the system generates two false positives for every accurate prediction.

Third, and most strikingly of all: the study was conducted almost exclusively on men — or rather, a dataset of veterans that is 93.6% male. Given the A.I…