American Research Group, Inc.

 

2014 SurveyMonkey Polling Accuracy

 

The following table lists the accuracy of SurveyMonkey polls in the November 2014 general election using the measure of polling accuracy proposed by Martin, Traugott, and Kennedy.

An accuracy measure of 0.00 indicates that the odds ratio of the survey results matches the odds ratio of the actual results (the closer the accuracy score is to 0.00, the better).

While SurveyMonkey claims it "accurately" predicted all 36 U.S. Senate races in 2014, the accuracy measures for 20 out of 36 senate races fall outside of the 95% confidence intervals. Also, the accuracy measures for 14 out of 35 gubernatorial races fall outside of the 95% confidence intervals.

Overall, the accuracy measures for 34 out of 71 races fall outside of the 95% confidence intervals. This represents nearly 48% of all the races polled. It would be expected that the accuracy measures for about 20 of the polls would fall outside of the 95% confidence intervals.

2014 SurveyMonkey Polling Accuracy
State Race SurveyMonkey Actual Results
 Odds Ratio Accuracy
Texas Senate 33% 59% 34.4% 61.6% 1.00 0.00 
Arizona Governor 40% 52% 41.6% 53.5% 0.99 -0.01 
Oregon Senate 55% 37% 55.6% 37.0% 0.99 -0.01 
Kentucky Senate 39% 53% 40.7% 56.2% 1.02 0.02 
Georgia Governor 41% 49% 44.9% 52.8% 0.98 -0.02 
New Mexico Governor 41% 54% 42.7% 57.3% 1.02 0.02 
Nebraska Governor 38% 57% 39.1% 57.4% 0.98 -0.02 
Florida Governor 43% 45% 47.1% 48.2% 0.98 -0.02 
New Hampshire Senate 49% 47% 51.5% 48.2% 0.98 -0.03 
Pennsylvania Governor 52% 44% 54.8% 45.2% 0.98 -0.03 
Idaho Senate 32% 62% 34.7% 65.3% 0.97 -0.03 
New Mexico Senate 52% 43% 55.5% 44.5% 0.97 -0.03 
Texas Governor 38% 56% 38.9% 59.3% 1.03 0.03 
New Jersey Senate 53% 39% 55.6% 42.6% 1.04 0.04 
Colorado Senate 44% 48% 46.2% 48.3% 0.96 -0.04 
West Virginia Senate 33% 62% 34.5% 62.1% 0.96 -0.04 
Colorado Governor 47% 46% 49.2% 46.1% 0.96 -0.04 
Rhode Island Governor 45% 42% 40.7% 36.3% 0.96 -0.05 
Georgia Senate 40% 49% 45.2% 52.9% 0.96 -0.05 
Ohio Governor 31% 63% 32.9% 63.8% 0.95 -0.05 
Kansas Governor 42% 48% 46.1% 50.0% 0.95 -0.05 
Iowa Senate 42% 53% 43.7% 52.2% 0.95 -0.06 
California Governor 54% 39% 59.4% 40.6% 0.95 -0.06 
Oklahoma Senate 2 27% 67% 29.0% 67.9% 0.94 -0.06 
Nevada Governor 23% 64% 23.9% 70.6% 1.06 0.06 
Maine Governor 38% 45% 43.3% 48.2% 0.94 -0.06 
South Dakota Governor 24% 71% 25.4% 70.5% 0.94 -0.06 
New York Governor 51% 36% 53.8% 40.6% 1.07 0.07 
Tennessee Governor 24% 69% 22.8% 70.3% 1.07 0.07 
Virginia Senate 47% 43% 49.2% 48.3% 1.07 0.07*
Arkansas Senate 37% 57% 39.4% 56.5% 0.93 -0.07*
Illinois Governor 43% 51% 46.0% 50.6% 0.93 -0.08*
Oregon Governor 51% 42% 49.8% 44.3% 1.08 0.08 
Oklahoma Senate 1 29% 63% 28.6% 68.0% 1.09 0.09 
Wyoming Senate 17% 77% 17.5% 72.2% 0.91 -0.09 
Kansas Senate 43% 49% 42.5% 53.2% 1.11 0.09*
Oklahoma Governor 42% 52% 41.0% 55.8% 1.10 0.10 
Alaska Senate 43% 51% 45.4% 48.6% 0.90 -0.10*
Louisiana Senate 36% 39% 42.1% 41.0% 0.90 -0.11*
Montana Senate 41% 53% 40.1% 57.8% 1.12 0.11*
Idaho Governor 42% 52% 38.6% 53.5% 1.12 0.11*
North Carolina Senate 41% 48% 47.2% 48.9% 0.89 -0.12*
Iowa Governor 34% 61% 37.2% 59.0% 0.88 -0.12*
Nebraska Senate 34% 62% 31.3% 64.6% 1.13 0.12*
Alaska Governor 48% 47% 53.7% 46.3% 0.88 -0.13*
Delaware Senate 50% 43% 55.8% 42.2% 0.88 -0.13 
Mississippi Senate 32% 59% 37.4% 60.4% 0.88 -0.13 
Minnesota Governor 47% 48% 50.1% 44.5% 0.87 -0.14*
Wyoming Governor 25% 63% 27.3% 59.4% 0.86 -0.15*
Illinois Senate 49% 46% 53.2% 43.0% 0.86 -0.15*
Michigan Governor 42% 53% 46.9% 50.9% 0.86 -0.15*
Massachusetts Governor 42% 51% 46.6% 48.5% 0.86 -0.15*
Maine Senate 34% 63% 31.6% 68.4% 1.17 0.16*
Arkansas Governor 37% 58% 41.5% 55.4% 0.85 -0.16*
Alabama Governor 30% 62% 36.4% 63.6% 0.85 -0.17*
Michigan Senate 49% 44% 54.6% 41.3% 0.84 -0.17*
Maryland Governor 49% 45% 46.9% 51.4% 1.19 0.18*
Connecticut Governor 44% 50% 50.8% 48.1% 0.83 -0.18*
Rhode Island Senate 64% 32% 70.6% 29.3% 0.83 -0.19*
South Carolina Governor 36% 59% 41.4% 55.9% 0.82 -0.19*
Massachusetts Senate 53% 40% 62.0% 38.0% 0.81 -0.21*
South Carolina Senate 1 29% 52% 38.8% 54.3% 0.78 -0.25*
Wisconsin Governor 40% 58% 46.6% 52.3% 0.77 -0.26*
South Dakota Senate 24% 53% 29.5% 50.4% 0.77 -0.26*
Minnesota Senate 47% 49% 53.2% 42.9% 0.77 -0.26*
South Carolina Senate 2 30% 65% 37.1% 61.1% 0.76 -0.27*
Vermont Governor 49% 36% 46.4% 45.1% 1.32 0.28*
Hawaii Governor 40% 43% 49.5% 37.1% 0.70 -0.36*
Hawaii Senate 58% 34% 69.8% 27.7% 0.68 -0.39*
Absolute average accuracy for all polls** 0.11  

*The accuracy measures for these 34 polls (approximately 48% of the total) fall outside of the 95% confidence intervals.

**For a comparison, the absolute average accuracy for the US Presidential race was 0.055 in 2012 with three out of twenty-five polls falling outside of the 95% confidence intervals and 0.037 in 2008 with none of the nineteen polls falling outside of the 95% confidence intervals.

Accuracy problems with the SurveyMonkey polls are found across sample sizes. Seven out of the ten polls with sample sizes of less than 1,100 have accuracy measures outside of the 95% confidence intervals and ten out of the fifteen polls with sample sizes of 4,000 or more have accuracy measures outside the 95% confidence intervals.

NBC News is using SurveyMonkey polls for the 2016 elections. Chuck Todd of NBC News says the SurveyMonkey methodology has been "rigorously vetted" and that the polls are "scientifically done." That may be the case, but the accuracy measures for the 2014 election show a different story.

ARG Home