August 26, 2010

Florida Primary Pollster Scorecard

Here are the pollster scorecard results for the Florida primary using the measure of polling accuracy proposed by Martin, Traugott, and Kennedy.

Democratic US Senate

The accuracy measures for the Susquehanna poll (0.21, with a standard error of 0.086), the Mason-Dixon poll (0.28, with a standard error of 0.126), the Quinnipiac poll (0.32, with a standard error of 0.108), the Feldman poll (0.38, with a standard error of 0.105), and the Ipsos poll (0.84, with  standard error of 0.182) fall outside the 95% confidence interval, showing biases toward Greene.

Republican Governor

The accuracy measures for the Tarrance poll (0.16, with a standard error of 0.079), the Mason-Dixon poll (0.29, with a standard error of 0.111), and the McLaughlin poll (0.38, with  standard error of 0.106) fall outside the 95% confidence interval, showing biases toward McCollum.

-- Dick Bennett

August 23, 2010

A total of 43% of Americans say they approve of the way Barack Obama is handling his job as president and 51% say they disapprove of the way Obama is handling his job. When it comes to Obama's handling of the economy, 41% of Americans approve and 54% disapprove.

Among Americans registered to vote, 44% approve of the way Obama is handling his job as president and 51% disapprove. On Obama's handling of the economy, 42% of registered voters approve and 53% disapprove. In the lastest survey, 87% of Democrats approve of the way Obama is handling his job. In July, 80% of Democrats approved.

While 49% of Americans say the national economy is getting worse, just 23% say their personal financial situations are getting worse. In July, 44% of Americans said the national economy was getting worse and 40% said their personal financial situations were getting worse.

Details from the nationwide survey conducted August 17-20 are available at The National Economy.

August 20, 2010

California Republican US Senate Primary Pollster Scorecard

Here are the pollster scorecard results for the California Republican US Senate primary using the measure of polling accuracy proposed by Martin, Traugott, and Kennedy.

Republican US Senate

The accuracy measures for the Field poll (0.44, with a standard error of 0.155), the USC/LATimes poll (0.46, with a standard error of 0.148), and the DailyKos/Research 2000 poll (1.48, with  standard error of 0.175) fall outside the 95% confidence interval, showing biases toward Campbell.

-- Dick Bennett

August 17, 2010

Connecticut Primary Pollster Scorecard

Here are the pollster scorecard results for the Connecticut primary using the measure of polling accuracy proposed by Martin, Traugott, and Kennedy.

Democratic Governor

Republican US Senate

Republican Governor

The accuracy measure for the Quinnipiac Democratic Governor poll (0.35, with a standard error of 0.107) falls outside the 95% confidence interval, showing a bias toward Lamont.

-- Dick Bennett

August 11, 2010

Colorado Primary Pollster Scorecard

Here are the pollster scorecard results for the Colorado primary using the measure of polling accuracy proposed by Martin, Traugott, and Kennedy.

Democratic US Senate

Republican US Senate

Republican Governor

The accuracy measure for the SurveyUSA Democratic US Senate poll (-0.24, with a standard error of 0.093) falls outside the 95% confidence interval, showing a bias toward Romanoff.

Update: In 2008, Public Policy Polling (PPP) had an average accuracy score of 0.018 for 30 state polls and SurveyUSA (SUSA) had an average accuracy score of 0.017 for 49 state polls (Table 3).

-- Dick Bennett

July 27, 2010

Martin, Traugott, and Kennedy Measure of Polling Accuracy

Using the measure of polling accuracy proposed by Martin, Traugott, and Kennedy with the results from the 178 polls contained in Nate Silver's spreadsheet for the American Research Group, our polls have an average odds ratio of 1.01 with a predictive accuracy of -0.03.

An odds ratio of 1.0 signals perfect agreement between the odds in the polls and the actual voting outcomes.

The margin of error for the measure of predictive accuracy at the 95% level is plus or minus 0.212. The measure for our polls of -0.03 is within the confidence interval, indicating that the polls are not biased.

The median of the odds ratio for our polls is 0.979. Placing this under the standard normal distribution shows an error of 0.008 (8/10's of a percentage point). This corresponds to the error calculated using the margin of error for the difference between the top two candidates.

Unlike the measure used by Nate Silver, this is a standardized measure that does not average in the margin of error as the number of polls increase and does not change based on the undecideds in the polls (higher undecideds translate into higher error in Nate's measure).

-- Dick Bennett

July 22, 2010

Nate Silver is wrong

When Warren Mitofsky was looking for ways to measure polling accuracy, he discovered that using the average absolute (unsigned) value of the difference between the polling results and the actual vote counts does not average out the sampling error as the number of surveys comprising the average increases.

This is not a problem when only a handful of polls comprise the average, but it does become a problem as the number of polls comprising the average increases. In his ratings, Nate fails to average out the sampling error by the number of polls conducted by each pollster. This is why Nate's average error for polls is much larger than the errors calculated by other researchers.

Nate's examples in his post that use results from 4 cherry-picked surveys to determine the average will not show the effect. The Law of Large Numbers suggests a minimum of 20 surveys are required in the average before the effect is noticeable. Nate does not acknowledge, nor correct for this, in his ratings.

Recognizing the limitations of measures of polling accuracy, Martin, Traugott, and Kennedy proposed a new measure for polling accuracy. Nate, however, uses, and as a result defends, the methods they criticize.

Perhaps Nate is not aware of this. His post, however, suggests the opposite because his spreadsheets for the pollsters he tracks would clearly show the effect Mitofsky discovered and he did not use those averages in his examples.

Nate is wrong about the impact this has on his ratings.

-- Dick Bennett

July 14, 2010

Update: Not So Much Additional Error

A subscriber to our e-mails writes:

I believe that Mr. Silver's average error rates are overstated because he forgets that the error range is + or - when he calculates the average error for each pollster. It's a common mistake.

If you have a poll where the difference between Candidate A and Candidate B is 9 and the actual difference between the two when the votes are counted is 4, the error is +5. But if you have a poll where the difference between Candidate A and Candidate B is 4 and the actual difference when the votes are counted is 9, the error is -5, not +5.

If he says the average error for both polls is +5, his average error will be overstated by the actual average margin. You should be able to check this using his spreadsheet.

I did check and it is true. Nate has a column on his spreadsheet labeled "error" which is the absolute value of the error (all positive numbers). The median value of this "error" for our polls is 5.08. The median error for our polls calculated based on our polls minus the actual results, however, is -0.54. When the actual margin of 4.54 is subtracted from Nate's median "error" of 5.08, the result is -0.54 (the correct value for our polls).

If Nate used the absolute value in calculating his pollster error, it makes a mess of his ratings because his error rate for each pollster is incorrect.

Time for v4.1 of the ratings.

-- Dick Bennett

July 13, 2010

Not So Much Additional Error

A commercial tracking client e-mailed asking about Nate Silver's new pollster ratings at fivethirtyeight.com. Knowing that we have developed tools to test our survey results, the client asked if we had evaluated Nate's database of our polling results using the tools. The client was concerned that Nate's new ratings put the accuracy of our results in doubt while she knew that tests of our tracking showed very little additional error ("pollster-introduced error" in Nate's ratings).

I looked at Nate's site and e-mailed her that Nate writes in his methodology statement that: "Technically speaking, our goal is not to evaluate how accurate a pollster has been in the past -- but rather, to anticipate how accurate it will be going forward (as these ratings are principally used in conjunction with our electoral forecasting)."

She e-mailed back the preceding sentence where Nate writes: "For instance, SurveyUSA's rawscore is -0.82, meaning that a SurveyUSA poll has about eight-tenths of a point less error than average. American Research Group's raw score is +0.72, meaning that it has seven-tenths of a point more error than average." She then wrote, "The rankings are based on past performance. What is the average? Is it significant? Also, he writes +0.72 for you and -0.82 for SurveyUSA, but his tables show +0.84 for you and -0.84 for SurveyUSA - which are correct?"

I e-mailed Nate and he kindly forwarded his spreadsheet for the American Research Group surveys contained in his database.

The short answer is that the average poll ranges in additional error beyond the theoretical margin of error from 0.47 to 0.93 percentage points based on our polling in Nate's database and using his scale from his tables (I am assuming the "rawscores" in Nate's tables are correct). Our additional error ranges from 0.8 to 1.6 percentage points. There are no significant differences between the average additional error and our additional error and no significant differences between SurveyUSA's additional error and our additional error.

So in a way, Nate's rankings are not based on polling accuracy because there are no actual differences in accuracy between our polls and the average and our polls and SurveyUSA, no matter which scale is used.

The reason for this is simple. The margin of error for the margin between the two leading candidates in a race, the measure Nate uses, can be up to twice the margin of error for the overall results. In a survey of 600 likely voters showing one candidate leading her closest opponent 46% to 41%, for example, the theoretical margin of error for the 5 percentage-point lead is plus or minus 7.4 percentage points, 95% of the time, while the margin of error for the survey is just plus or minus 4 percentage points. 

There are two easy ways to see this in Nate's data of our polling. The first is to look at the deviation of our results from the actual results based on the margin of error for the differences in proportions of the top two candidates. 

The median value of our polling margin (because we are going to place the results under the standard normal distribution), labeled "margin_poll" in Nate's spreadsheet, is 4.00 and the median value of the actual margin from the election results, "margin_actual" in Nate's spreadsheet, is 4.54. That is a difference of -0.54 percentage points (4.00-4.54=-0.54).

The median value of our polls in Nate's spreadsheet for "cand1dem" is 45.0 and the median value for "cand2gop" is 43.0. The median sample size for our polls is 600. Using our Ballot Lead Calculator with those values, the margin of error for the difference between candidates is plus or minus 7.5%, 95% of the time (half of the reported confidence interval). That margin of error is placed under the standard normal distribution by dividing by 1.96 (95% confidence level), giving 3.8 (7.5/1.96=3.8). We are interested, however, only in plus or minus 1 standard deviations under the standard normal distribution so the 3.8 is multiplied by .6826, giving 2.59. This covers over two-thirds of cases and the difference of our average margin (-0.54) should fall within these boundaries (-2.59 to +2.59), which it does.

When our difference (-0.54) is placed within plus or minus 1 standard deviation of the standard normal distribution, the result is a z-score of -0.21 (-0.54/2.59=-0.21). This z-score translates into a margin of error of plus or minus 8.3 percentage points, 95% of the time. The difference between the theoretical margin of error and our actual error using this test is 0.8 percentage points (8.3-7.5=0.8). That is, the margin of error for our margin results in Nate's spreadsheet is about 1 percentage point higher than the theoretical limit. This is a measure of pollster-introduced error.

The second way to test our results is to calculate a z-score using the adjusted median absolute deviation for the actual voting margin, which is 13.56 based on Nate's spreadsheet. The z-score of -0.04 is calculated by subtracting our polling median from the actual margins and dividing by the deviation (4.00-4.54/13.56=-0.04). This z-score translates into additional error of plus or minus 1.6 percentage points. This is another measure of pollster-introduced error.

Using our higher pollster-introduced error of 1.6 percentage points and Nate's scale in his charts, the error for SurveyUSA would be 0.14 percentage points, the average error would be 0.87 percentage points, and the error for our surveys would be 1.6 percentage points. These differences are not statistically significant.

The error rate for our lower pollster-introduced error of 0.8 would be 0.07 percentage points for SurveyUSA, 0.44 percentage points for the average, and 0.8 percentage points for our surveys. These differences are also not significant.

Nate claims the average overall additional error rates in his database range from 2.8 to 7.8 percentage points based on the type of race polled. Those error rates are higher than we found using Nate's spreadsheet for our results, so the estimates above may be off (see the Update above - Nate's methodology overstates the error because it includes the average margin for actual results in the races polled in the error). Unlike Nate's rankings, however, we did not weight our calculations by the "recency" of the polls, the type of election, the race involved, or award a bonus to some pollsters.

While Nate states his ratings are "objective," the bonus he gives to "polling firms which have made a public commitment to disclosure and transparency" because the scores of those firms "hold up better over time" does not support that claim. Nate bases his claim on a regression analysis and that the variable he used ("ncppaapor") "is statistically significant at approximately the 95 percent level."

Looking at the output Nate provides, the variable may be close to being significant, but the total variance explained (R-squared) is just slightly over 4%. This is an unconditional regression with a very low r-squared and his conclusion does not appear to be justified.

Mark Blumenthal at pollster.com put together a spreadsheet that removed the "bonus" Nate provided and 23 of the 25 firms given the bonus dropped in ranking. The average drop was 90 places. The largest drop was for PSRA, which went from 27th place in Nate's ranking to 204th place when the bonus was removed. If Nate were correct in saying that the bonus was justified because the scores hold up better over time, then the firms receiving the bonus should have held their places. They didn't.

I'm not exactly sure what Nate is measuring in his rankings, but I thank him for providing the spreadsheet that shows our polling with actual pollster-introduced error of between 0.8 and 1.6 percentage points for a variety of political contests.

-- Dick Bennett

Why doesn't most advertising work? Take a look at 10 Rules for More Effective Advertising.

See your out-of-conscious processing system at work and learn how to improve your marketing and advertising efforts by understanding the power behind the Scintillating Grid.

The margins of error reported for most polls underestimate the actual margins of error for ballot estimates from the same samples. Use the Ballot Lead Calculator to determine if the lead for any survey is statistically significant.