Regardless of your political preferences or feelings about the outcome of the 2016 US presidential election, all sides agree that it was a stunning conclusion to one of the strangest and most acrimonious elections in most voters’ lifetimes. While most of us are breathing a collective sigh of relief that it’s over, one aspect worth analyzing from an IT leadership perspective is how polling completely missed the outcome.
The elusive crystal ball
Polling is fundamentally an exercise in predictive analytics, and despite the political bent, broadly follows a process and leverages enabling technologies that should be familiar to any leader. Much like everything from product recommendation engines, to forecasting and modeling tools used by businesses, pollsters create mathematical models based on historical data and inputs from the political equivalent of “customer sentiment analysis” through voter surveys. Much like the evolution in analytics software and techniques, with each election we get fancier visualizations, more advanced models, and an increasing sense that we’ve been handed a crystal ball that can predict the future with relatively high accuracy. In fact, the few polls that predicted a slim Trump victory were roundly and loudly criticized as the press proclaimed that Trump had a “ridiculously narrow path to victory.”
The fact remains that all the best systems, visualization tools, and analytical models in the world are not a guarantee of your ability to forecast the future. Despite the word “predictive,” analytical models are based on the past, and use assumptions based on the past to guess at the future. For most complex systems, from the weather to consumer buying behaviors, the “rules of the game” are subject to rapid change that can thwart even the best analytical tools.
The question problem
The emerging story around what “went wrong” is that pollsters erred primarily on two fronts. The first was an ineffective process for determining who was a “likely voter.” Most polls took an apparently logical approach and asked their survey pool whether they were planning to vote or not. Like any survey, whether related to an election or buying a consumer product, the customer’s answer doesn’t always reflect reality.
I’ve seen this dramatically demonstrated when consumers were asked, while standing next to their freezer, if they ever froze fresh meat for later use. Many insisted they would never do such a thing, as they strongly preferred fresh meat, yet upon opening the freezer, were “shocked” to see they’d frozen several packages of meat and revised their unqualified “never.” There are numerous psychological studies and theories around why this is the case, but essentially we often have an idealized “mental model” of ourselves, and when asked to describe ourselves we convey that idealized model rather than reality. The poll that came closest to predicting a Trump win asked voters to rate their likelihood of voting on a 100-point scale, allowing a degree of nuance that let the poll designers more accurately predict who would vote versus a simple yes or no question, where that idealized mental model apparently was far more likely to vote than the actual person.
The other big miss cited in the election was with exit polling, which predicted a comfortable Clinton win early in the voting process. Intriguingly, it appears Clinton voters were more likely to speak with exit pollsters, so exit polling results were skewed in her favor. Imagine performing a survey on a crowded street and asking the question “Do you enjoy taking surveys?” Your results would obviously be wildly biased since the people most likely to respond would inherently enjoy taking surveys. No one thought to question whether voters would be willing to talk to exit pollsters in equal numbers, or whether people would accurately convey how they voted.
Avoiding your own data debacle
Intuition may seem quaint in the face of petabyte data lakes, predictive analytics, and a team of mathematics PhDs, but it’s a key element in predicting the future, especially when your underlying assumptions could deviate from reality, or reality could rapidly change as environmental conditions change. Furthermore, surveys and marketing analytics firms may provide an easy way to gather large amounts of customer sentiment, but it’s worth actually observing and speaking with some of your customers. Many in the press who penned the headlines about an assured Clinton win are now suggesting they spent minimal time among different demographics and geographies, essentially letting the “echo chamber” of large urban areas, and their own personal preferences, convince them that a Clinton win was assured.
Even if all the data in the world indicate your next product will be a hit, break away from the focus group videos and spreadsheets and talk to potential customers face to face. Get out of the office and observe how they work and would use your product, and see if your models and assumptions are built upon reality, or your own invalid version of reality.
AI tool successfully predicted Trump win; still, experts are skeptical
TechRepublic’s ‘swarm AI’ predicts tight election, gives edge to Clinton
Political betting markets show late momentum shift towards Clinton