Can Google help you to diagnose patients' problems?

This month Trish Groves takes you through a cross sectional study that tried to find out whether the internet search engine Google is useful as a diagnostic tool

csiro
Dr Google will see you now

Why did the authors do the study?

One of the doctors who did this study saw a teenager whose father had used Google to find a diagnosis: "After evaluating a 16 year old water polo player who presented with acute subclavian vein thrombosis, one of us (HT) started to explain that the cause of the thrombosis was uncertain when the patient's father blurted out, "But of course he has Paget-von Schrötter syndrome.'"

The father had used Google to search the web for information and, on the strength of this, gave the doctor a mini-tutorial on the pathophysiology (hypertrophy of the neck muscles leading to dynamic compression of the axillary vein at the thoracic inlet-leading to thrombosis) and treatment of the syndrome. This made the doctor ask, "How good is Google in helping doctors to reach the correct diagnosis?"

The answers to this question could be "potentially confusing" or "overwhelming," given that, when I put the term "acute subclavian vein thrombosis" into Google's search box on 29 November, the search yielded 103 000 results. How can a doctor, let alone a patient without medical training, evaluate so much information? Tang and Ng set out to tell us.

What did the researchers do?

The authors decided to use previously published case reports to test Google's ability to find diagnostic information. They looked at reports published in one journal, the New England Journal of Medicine, in 2005 and chose all 26 reports that focused on diagnosing rather than managing illnesses. These were mostly "mystery illnesses"-complex diseases with non-specific symptoms or common diseases with rare presentations.

From each report they picked the three to five main words or phrases that most specifically described the patient's symptoms and signs and entered them on to a data sheet (the words and phrases the authors chose are given in a table on bmj.com).They didn't read the reports further to learn the final diagnoses.

The authors used these words and phrases as search terms on Google and, for each case report, noted the three most prominent and appropriate diagnoses that Google found. To save time-and, perhaps, knowing that Google yields the best websites first (by ranking web pages in order of the number of links to them: a reasonably good proxy for quality)-they looked only at diagnoses mentioned in the first 30 search results.

When the authors compared these diagnoses with articles reported in the New England Journal of Medicine their internet searches had found the correct diagnosis in 15 out of 26 cases (58%). For example, from a case report of a 48 year old man with loss of consciousness while jogging, Tang and Ng chose the search terms "cardiac arrest", "exercise", "young", and Google found articles indicating a diagnosis of hypertrophic obstructive cardiomyopathy, which turned out to be correct for that case. On the other hand, for a report of a 17 year old tsunami survivor with respiratory distress and paralysis (hemiplegia) on the right, the Google search terms were "pulmonary infiltrates" and "CNS (central nervous system) lesion": the internet came up with a diagnosis of aspergillus (fungal) infection while the patient had actually had pneumonia and a brain abscess from breathing in water contaminated by several types of bacteria.

Did this study show whether Google was a useful diagnostic tool?

Google found the right diagnosis in 58% of cases. Tang and Ng reported that this result had a 95% confidence interval of 38% to 77%. To put it another way, the true percentage had a 95% chance of falling somewhere between these two percentages if someone did the same study many times with another set of 26 case reports from the same journal in another year (if all other things stayed the same, for instance, the journal's criteria for accepting case reports and the authors' criteria for selecting search terms and picking the best diagnoses). This is a wide interval, and one that left some BMJ readers unimpressed. Of the doctors who sent a rapid response to bmj.com, Oltunde A Ashaolu, said "Tang and Ng seem to be comfortable with a correctly predicted diagnosis of 58% (with a lower confidence interval limit of 38%) whereby a lot of my colleagues may feel much less so The researchers have expertly selected relevant symptoms to input data into Google and a yield of 58% is obtained. A junior doctor may input a slightly different set of symptoms and the array of available information in fact becomes a quagmire of confusion rather than being an educative medium."

Was this a good study?

The study used a simple cross sectional design, looking at and describing a clearly defined sample (of case reports) in two different settings-firstly in a journal and then, after reducing the case reports to just a few key words, in the context of the web. Cross sectional studies are most often used in medical research for surveying a population or sample of patients at one point in time to see what health problems or risk factors they have. They cannot tell you anything meaningful about the causes of an illness or the effects of an intervention or experiment.

It would be going too far to describe Google as a diagnostic test in this study; at best it's a tool. The equivalent diagnostic test would be: the whole package of search terms chosen by doctors with some clinical expertise plus the use of the search engine plus the contents of the web pages the search found plus the doctors' criteria for selecting the best pages. This test, therefore, relies a lot on the doctors' expertise and knowledge. And to assess properly the performance and value of such a diagnostic test the authors would have had to compare their Google method with the best diagnostic test currently available-the "gold standard" and calculated both of the tests' sensitivity (ability to correctly find the right diagnoses) and specificity (ability to spot the wrong diagnoses). Comparing Google with another search engine would have been a good start, and it would have been most useful and valid to compare Google with the widely used academic search engine PubMed.

Indeed, one rapid responder to the BMJ, Reinhard Wentz, did just that and found that a PubMed search found the right diagnosis 88% of the time, for the same case reports Tang and Ng used and with the same search terms. It took him about three minutes per case report. Dr Wentz also said, "Is Google perhaps the modern version of 10 000 monkeys sitting at 10 000 typewriters, who, given time, will eventually produce a true copy of Hamlet? Among the one correct version there will of course be thousands with just one or two typos. So, can Mark Twain's remark 'Be careful about reading health books. You may die of a misprint' be applied to Google?"

If Tang and Ng had wanted to assess the validity of their diagnostic test, an even better gold standard would comprise all of the best current diagnostic methods for each of the conditions described in the 26 case reports. This would have been much more clinically meaningful and scientifically strong. Another rapid responder, Catherine A Smith, also pointed out that the study would have been much more useful if the researchers had given us information about the sources of the diagnostic answers-both the correct and the incorrect ones.

If Google had found the right diagnoses more often, with less need for expert medical knowledge on the part of the researchers, the article might have prompted even more debate on bmj.com about whether patients really need doctors any more.

How much does this study matter?

I would give Tang and Ng's study six out of 10 for clinical relevance and scientific strength because it was only cross sectional and did not compare Google with any other way of finding diagnoses. But the journalistic merits of the work, its topicality, and its success in prompting debate have bumped up my overall score to eight out of 10.

Tang and Ng wanted to assess, in a reasonably objective way, something that many patients and doctors already do. For instance, they say that 25 million people in the United Kingdom alone had access to the web by 2001, and that searching for health information is one of the commonest uses of the web. Given that doctors are expected to remember about two million facts, this small and preliminary study was worth doing and is worth reading.

Trish Groves, senior editor, studentBMJ
Email: tgroves@bmj.com


studentBMJ 2007;15:1-44 January ISSN 0966-6494