Predicting Disease with Google & Comets

Google Flu has been getting a lot of coverage lately, mostly for how it is getting the flu “wrong.” There was a Nature news item in February about Google Flu overestimating the peak of the current flu season. Not to be outdone, Science followed up a month later with an article on the forecast errors of Google Flu from the past two seasons as well as the current one. (original paper here) Also mentioned in the Nature item is the fact that Google Flu missed the initial novel H1N1 epidemic in the spring of 2009.

While there is no denying the mismatch between Google Flu’s numbers and the actual observations of flu incidence, I’m not sure I’d say that Google Flu is wrong so much as I would submit that we might be asking the wrong questions of it. Google is surely correct about the numbers of people searching for different keyword terms. And the correlations it has observed between those numbers and the incidence of flu are presumably genuine correlations. But that doesn’t mean we can expect that data to be able to answer questions about how many people will get the flu next week.

The recent episode of Cosmos, entitled “When Knowledge Conquered Fear,” illustrated the problem of correlation and prediction nicely with the story of comets and how our understanding of them has changed over time. Originally, comets were thought to be portents of calamities to come. Of particular note for public health, some cultures believed comets to be an omen of smallpox.

One imagines that this belief was based on genuine empirical data. It seems quite plausible that multiple comet sightings coincided with cases or outbreaks of smallpox. Sure,  maybe some confirmation bias played a role in weighting certain data points. And yes, the time lag between the two may have been treated less rigorously than we might have preferred; call it a “heuristic.” Nevertheless, I wouldn’t be surprised if there were a genuinely measurable correlation in the data at some point, even though the comet-watchers were drawing their inferences millenia before Pearson and p values.

And yet, nowadays we don’t use comets to predict smallpox or any other communicable diseases, no matter how convenient that might be for epidemiologists. At some point, the correlation broke down. The smallpox-comet model probably faded away slowly; one can almost hear the public health astronomers saying something similar to John Brownstein (quoted in the Nature item): “You need to be constantly adapting these models, they don’t work in a vacuum. You need to recalibrate them every year.”

But as the narrative of the show explains, the correlations between comets and calamities were eventually set aside in favor of correlations with orbital periods. Those correlations continue to be of interest, since they allow for accurate predictions of when comets will be seen again. And they are predictive because they are more closely related to the causal pathway that leads to comets becoming visible in the sky. Whereas there does not appear to be any causal link between comets and smallpox.

I think it’s fairly safe to assume that using Google to search for particular keywords does not cause many cases of the flu; keyboards and phone screens may be vectors for the virus, but that should be largely independent of the words input using them. So that leaves the question of whether being infected with the flu is a cause of particular keyword searches, or if there is a more indirect causal relationship. Media coverage is often proposed as a causal factor in searches; people may also be moved to search if a friend or relative is sick, or if flu is rumored to be “going around” at school. (Armchair epidemiology is a popular topic at my kids’ bus stop.) Media coverage itself could be caused by actual cases and also by vaccine shortages or public health prevention campaigns. And so on.

With a better understanding of the causal connections, we can begin to ask better questions of the Google data. Perhaps we should be using it as an indicator of public concern, or public awareness. If public health is trying to get a particular message out specific to the current flu season, we might be able to gauge how well that message has been taken up by the public by how often they search for a particular word or phrase in that message. Maybe we can ask questions about what is unique for the current season compared to previous ones in terms of word usage in flu searches.

Meanwhile, we already understand the causal connections between the flu and other things we can measure. Having the flu causes people to have fevers, which causes them to buy thermometers, which is something we can and do measure and have observed to be correlated with flu cases. Having a fever also causes people to visit their primary care physician, an urgent care center, or the emergency department; visits motivated by fever and other flu symptoms are another thing we can measure and which are correlated with flu cases. As long as the flu causes the same symptoms, these metrics will always be useful predictors.

Google will no doubt be able to continue to identify more specific correlations that can improve the fit between their data and the CDC flu numbers, at least for the seasons that have already occurred. But if we continue to expect it to predict the flu without a better understanding of the causal connections between illness and searching, and without knowing what questions Google’s data can and cannot answer, I predict we haven’t seen the last “Google Flu got it wrong”-style headline.