American Research Group, Inc.




Conflicting state and national polls for the 2020 Democratic presidential nomination are a common and ongoing occurrence. The polls illustrate the substantial theoretical limitations of survey sampling. Pollsters, poll sponsors, and polling aggregators tend to ignore those limitations in favor of assigning a level of accuracy to the polls that are unjustified and unrealistic. The 95% confidence interval reported by pollsters is as misleading as it is misunderstood. Here is an example:

In late August, Monmouth University released the results from a national survey among a subsample of 298 of respondents identifying themselves as Democrats or leaning Democrats showing Elizabeth Warren and Joe Biden tied in the race for the Democratic presidential nomination at 20% each and Bernie Sanders at 16%. Biden had dropped from 32% in a Monmouth poll completed in June while Warren was up from 15% and Sanders was up from 14% compared to that June survey.

At the same time, a nationwide YouGov survey including a subsample of 559 Democrats had Biden at 22%, Sanders at 19%, and Warren at 17%. Four polls released later in the week, however, had Biden up by 13 points in three of the polls and Biden up by 18 points in the other poll.

In response to this, Patrick Murray, the director of the Monmouth University Poll, released a statement that included the following:

As other national polls of 2020 Democratic presidential race have been released this week, it is clear that the Monmouth University Poll published Monday is an outlier. This a product of the uncertainty that is inherent in the polling process. We tend to focus on the margin of sampling error, but that margin is driven by something called the confidence interval which states that every so often you will naturally have a poll that falls outside the standard margin of error. It occurs very infrequently, but every pollster who has been in this business a while recognizes that outliers happen. This appears to be one of those instances.

Nate Silver followed up on with an article titled "How to Handle an Outlier Poll." He writes, in part:

But Murray doesn’t have any real reason to apologize. Outliers are a part of the business. In theory, 1 in 20 polls should fall outside the margin of error as a result of chance alone. One out of 20 might not sound like a lot, but by the time we get to the stretch run of the Democratic primary campaign in January, we’ll be getting literally dozens of new state and national polls every week. Inevitably, some of them are going to be outliers. Not to mention that the margin of error, which traditionally describes sampling error — what you get from surveying only a subset of voters rather than the whole population — is only one of several major sources of error in polls.

Murray and Silver provide nonsensical technical definitions of outliers. It is impossible for the results of a poll to be outside of its own margin of error. It is easy enough to know what they mean by an outlier — simply a poll that differs from other polls. But, more importantly, they are incorrect about the frequency of outliers in individual election polls.

Sampling theory is based on
"frequently and independently" repeating the same survey. Using the sample size and the results from each repeated sampling, margins of error and the resulting confidence intervals are calculated for each sample. As the sampling is repeated using the same sample size and methodology, the theory states that the population values should be contained within the confidence intervals very close to the desired level of confidence, usually 95% of the time for public polls, if an infinite number of samples are drawn.

Jerzy Neyman pointed out when introducing the concept of confidence intervals that frequentist probability theory is "
helpless" in providing the true population values from any single sample because the values obtained from the single sample provide no information about the actual population values. Here is a demonstration of sampling error at work. Note it is unlikely that any of the 50 samples exactly match the population values. The average of the 50 samples is more likely than any single sample to match the population. This is why polling aggregation works.

Pollsters generally incorrectly assume that not only their poll results reflect the actual population values (which the sampling error demonstration proves to be untrue), but also the population results fall within the the confidence intervals 95% of the time for a single poll. This is why Murray writes that outliers happen "every so often" and "very infrequently" and Silver puts outliers happening at "1 in 20 polls" (or 5% of the time).

In fact, actual outliers should occur about
25% of the time for individual election polls, or about 1 in 4 polls.

While it would seem that an event with a 95% probability of occurring in the long run would also be very likely to occur in a single event, this is not the case. Calculating for the worst case shows that an event with a 95% probability of occurring in the long run has a worst-case uncertainty of about 29% for a single event, not 5%. It takes an event with a 99.5% probability of occurring in the long run to have a worst-case uncertainty of 5% for a single event. (Hard to believe, but an event with a 71.4% probability of occurring in the long run has a worst-case uncertainty of about 86% for a single event.) 

This can be tested empirically by comparing election polls to election results.

Our election polling accuracy ratings for over 5,000 final election surveys from 45 pollsters show on average that 75% of election poll results are within the 95% confidence intervals when compared to the actual vote totals (the population values), confirming the theory for individual polls. (This paper has election polls within the theoretical margins of error 73% of the time for senatorial polls, 74% of the time for gubernatorial polls, and 88% of the time for presidential polls.)

Monmouth University election poll results have been within their respective theoretical margins of error about 83% of the time, which is above average. But that still means about 1 in 6 election polls from Monmouth fell outside their respective theoretical margins of error when compared to the actual election outcomes.

Our accuracy ratings include sample sizes, which allow the accuracy of any poll to be compared to the accuracy of any other poll. The smaller the sample size of a poll, the wider the confidence interval, and, conversely, the larger the sample size of a poll, the narrower the confidence interval.

Monmouth polls tend to have smaller sample sizes, and, therefore, wider confidence intervals (note the sample size of just 298 Democratic primary voters in the national survey above).

In the 2016 presidential race in Wisconsin, for example, the final Monmouth poll was 47% for Clinton and 40% for Trump with a sample size of 403. When the poll results are compared to the actual Wisconsin results of 46.45% for Clinton and 47.22% for Trump, the Monmouth poll results were within the poll's theoretical margin of error with an accuracy score of 0.18. The final Marquette Law School poll in Wisconsin had Clinton at 46% and Trump at 40% with a sample size of 1,255. The results from the Marquette Law School poll, because of the larger sample size, fell outside of that poll's theoretical margin of error, but the poll had an accuracy score of 0.16. The Monmouth poll in Wisconsin was not very accurate, but the results were what could be expected based on sampling theory with such a small sample size. The Marquette Law School poll was more accurate than the Monmouth poll, but it should have been even more accurate based on its larger sample size.

For elections from 1978 through 2018, results from most pollsters perform as sampling theory predicts. Looking at all 45 pollsters and constructing a 95% confidence interval for accuracy by pollster shows that the results from only 4 pollsters fall outside of that 95% confidence interval. They are Harris Interactive, the Trafalgar Group, SurveyMonkey, and We Ask America. In election polling in 2014, for example, SurveyMonkey polls were outside their respective theoretical margins of error for the actual vote about 48% of the time and that trend continued in 2016.
Silver writes that sampling error "is only one of several major sources of error in polls." Sampling error can be determined from the survey data. In the polls we have tested, the average non-sampling error accounts for less than one-third of all error (including "house effects"), making sampling error the major source of error in those polls. This is reflected in our polling accuracy scores. The average absolute accuracy for the 75% of election polls that fall within their respective 95% confidence intervals is 0.08 while the average absolute accuracy for the 25% of election polls that fall outside their respective 95% confidence intervals is 0.29. Unfortunately, sampling error can be substantial, but there is no way to control for sampling error.

As for Monmouth's election polling accuracy, the average absolute accuracy for the 83% of Monmouth University election polls that fall within their respective 95% confidence intervals is 0.09 and the average absolute accuracy for the 17% of election polls that fall outside of their respective 95% confidence intervals is 0.34.

When looking at 2020 election polls, remember that (1) the results from about 1 in 4 polls on average will fall outside the respective confidence intervals for the population values based on sampling theory alone, (2) the average of all polls should be a better indication of the actual state of the race than any single poll, and (3) the results from most pollsters should be included in polling averages.

Update: Election polls from the University of New Hampshire have been within their respective theoretical margins of error about 72% of the time, which is below average. The average absolute accuracy for the 72% of University of New Hampshire election polls that fall within their respective 95% confidence intervals is 0.11 and the average absolute accuracy for the 28% of election polls that fall outside of their respective 95% confidence intervals is 0.33.

ARG Home