How the Margin of Error Skews Our Perception of Risk

“Like Virginia, Florida remains too close to call. There Obama leads Mitt Romney at just 48 to 47 percent–less than the polls margin of error of plus or minus 3 percent.”

Virginia race too close to call, CBS Virginia

If you’re following the election polls, you’re accustomed to hearing a reporter announce the poll results along with a margin of error, and then telling you whether or not the results are statistically significant.

Their use of the phrases margin of error and statistically significant is mathematically accurate but highly misleading.

Margin of error measures one very specific type of risk: the chance that my poll sample is inaccurate assuming that I succeeded in:

• Accurately measuring
• A randomly selected
• Representative sample
• Of actual voters

In that case, the margin of error tells me I have 95% confidence that the actual result falls within the margin of error.

We focus on margin of error for the same reason the drunk in the joke looks for his keys under the lamppost. Because it’s easier than looking where the keys are likely to be. It’s an easy number to calculate. But it’s not where the main risk is. And citing the margin of error glosses over the bigger sources of risk, and gives us false confidence in the result.

Let’s take each part:

• Accurate measurement & the social desirability bias: If a priest told you that he polled his parishioners and they all give charity and none use contraceptives, would you believe him? Some people skew their answers to what they think the asker will like and respect. That’s why gay marriage usually wins in polls even though it has yet to pass in an election. The demonization of the right is unusually strong this year. How many people are telling the pollsters that they’ll vote for one candidate and then either not vote, or vote for the other guy?
• Random, representative sample: The art and science of gathering a representative sample contains the following potential sources of error:
  • Initial sample: Pollsters used to randomly select from a list of registered voters. Now they call people and ask if they’re registered. Cellphone only voters have introduced a more recent complication. The New York Times quoted an expert who said “Anyone who claims there’s a best practice doesn’t know what they’re talking about. We as an industry don’t know.”
  • Sub-sample: Over 90% of people called refuse to participate. Are the fewer than 10% who do participate representative of those who refuse?
  • Weighting the results. Pollsters then try to weigh the results to match their expectations of the turnout. For example, a few weeks ago the Gallup poll gave a few points bump to the president when they decided to decrease the weighting on white voters. How accurate is their projection for turnout model by demographic?
• Actual voters: Only about 56% of American adults vote. The 44% of adults who don’t vote are predominantly Democrat. What percentage of them pass through the polls’ likely voter screens? Jay Cost calculated that the last Washington Post poll included 73% of adults as likely voters. Does that skew the results?
• Statistical variance (aka margin of error): Even when I accurately measure a perfectly random and representative sample, I can be off by a few percent.

Margin of error refers only to that last point. We can argue about the details. The point is that the statistical margin of error is a drop in the bucket compared to all the other potential sources of error. It’s conventional wisdom to say that we know that this election will be close. I know no such thing. I will not be shocked by either candidate clearing 300 electoral votes. I strongly suspect that if there is a large gap between reality and the polls, the media and pollsters will defend their honor by blaming the gap on either fraud or on some late breaking phenomenon, like the hurricane fallout or the Benghazi investigation.

Corporate context

“What looks like tomorrow’s problem is rarely the real problem when tomorrow rolls around.”

James Fallows, Future Babble

One way companies deal with risk is on disclosure documents to investors. They often list every imaginable risk. Something like: if a nuclear holocaust exterminates humanity, our quarterly revenue may be adversely affected. I’m guessing that most of these statements are seen only by the lawyers who draft them and the auditors who review them.

Some companies list risk factors on internal strategic documents that are treated much the same way. Some managers like them because they give hard numbers, and the illusion of knowledge and control. These numbers are often not actually used for anything, or believed by anybody making operational decisions.

Then there are the factors that we actually concern ourselves with.

In measuring risk:

• Think outside of the box with a people from different departments and with different experience to consider where the serious risks actually are.
• Don’t worry at first about how you’re going to quantify the risk. Focus on quantification may bias you to focus on the risks that you can most easily quantify instead of the ones most likely to destroy your business.
• Don’t get lost in the science and math, or fooled by descriptive names. Margin of error is a mathematically precise phrase, but it ignores most of the actual sources of error. Pay close attention to all the sources of risk and error that your measurements don’t consider.

This is the part of a series of posts about management lessons from the US elections.
Previous: Only 3% of Obama Supporters or 17% of Romney Supporters Are Right, Reputation Management Lessons from You Didn’t Build That