2014 NFL Prediction Accuracy

Microsoft's Cortana and Nate Silver's fivethirtyeight.com Elo are predicting the outcomes of NFL games each week during the 2014 season.

The tables below use Brier scores to determine the accuracy of the probabilistic predictions from both models.

Brier scores range from 0 to 1, with 0 meaning the probabilities in the models perfectly match the outcomes of the games and 1 meaning no matches were made (so the closer to 0.000, the better calibration of the model).

If a 50% (0.500) probability were assigned to each team for each game, the Brier score would be 0.250.

For the season, the Elo model has a Brier score of 0.208 and the Cortana model has a Brier score of 0.213.

For the season, the Elo model performed about 17% better than assigning all teams a 50% win probability and the Cortana model performed about 15% better than assigning all teams a 50% win probability. The Brier scores for the season reflect that the average win probabilities among winning teams are 58% for the Elo model and 57% for the Cortana model.

Both models, however, failed to perform better than assigning each team a win probability of 50% in the second week of the season. These performances in week 2, worse than guessing, invalidated both models and the Brier scores show no improvements were made to the models over the remainder of the season.

Both models are low performing. This is why Neil Paine of fivethirtyeight.com wrote on September 24 that the results from the Elo model should be used for "entertainment purposes only."

On a weekly basis, the models failed to perform better than assigning each team a 50% win probability eighteen out of twenty-one weeks.

Another way to evaluate the accuracy of the probabilistic predictions of each model is to construct reliability tables. Predictions are placed in probability bins based on the probabilities of the winning team.

For example, 20 out of 79 teams (25%) won games where Cortana predicted a win probability between 20% and 29.9%. The target for this probability bin is 25% and Cortana predicted wins 25% of the time. Looking at the same probability bin for the Elo model, 11 out of 55 teams (20%) won games when the Elo model predicted a win probability between 20% and 29.9%.

The absolute difference between the target probabilities and the actual probabilities of the two models show the Cortana model, while having a slightly higher (worse) Brier score than the Elo model, performing better than the Elo model in the reliability tables.

The Brier score for the Cortana model for the season is due to a lower average predicted win percent (66%) among winning teams, compared to an average predicted win percent of 68% for the Elo model for winning teams. 

In theory, because the win probabilities for both models are calculated by simulating each game thousands of times, the target and actual probabilities should match for an absolute difference of 0.

When teams should have won 75% of the time with the Cortana model, they won 74% of the time. When teams should have won 85% of the time with the Cortana model, they won 83% of the time.

When teams should have won 75% of the time with the Elo model, they won 80% of the time. When teams should have won 85% of the time with the Elo model, they won 81% of the time. The Elo model was more prone to over and under-forecasting, indicating a bias in the model.

Also, 182 (68%) of Cortana's favorites won games compared to 180 (67%) of Elo's favorites winning games.

Finally, the Cortana model predicted that the New England Patriots would win the Super Bowl while the Elo model predicted the Seattle Seahawks would win the Super Bowl.

As a point of comparison, fivethirtyeight.com's Brier score was 0.1969 over 67 games in the 2014 NCAA Men's basketball tournament. The model for the NCAA Men's tournament performed 21% better than assigning each team a 50% win probability, but the Brier scores were greater than 0.250 in 3 of 6 rounds (including the first round), therefore invalidating the model. The Cortana model did not predict the NCAA Men's tournament.

Weekly Results

In the championship round, the Elo model had a Brier score of 0.088 and the Cortana model had a Brier score of 0.179.

This is only the third week of the season where the models performed better than assigning each team a 50% win probability. It should be noted that the original Cortana prediction for Seattle was 0.467, which would have produced a Brier score of 0.236 for the week. 

In Wild Card Weekend, the Elo model had a Brier score of 0.225 and the Cortana model had a Brier score of 0.233.

The 95% confidence intervals for both models for Wild Card Weekend include 0.250, making this the sixteenth week that both models did no better or worse than assigning each team a 50% win probability.

In week seventeen of the season, the Elo model had a Brier score of 0.208 and the Cortana model had a Brier score of 0.208.

The 95% confidence intervals for both models for week 17 include 0.250, making this the fifteenth week that both models did no better or worse than assigning each team a 50% win probability.

In week sixteen of the season, the Elo model had a Brier score of 0.257 and the Cortana model had a Brier score of 0.258.

In week 16, both models performed worse than assigning each team a 50% win probability. This the fourteenth week that both models did no better or worse than assigning each team a 50% win probability.

In week fifteen of the season, the Elo model had a Brier score of 0.155 and the Cortana model had a Brier score of 0.150.

This is only the second week out of 15 that both models performed better than assigning each team a 50% win probability.

In week fourteen of the season, the Elo model had a Brier score of 0.218 and the Cortana model had a Brier score of 0.237.

The 95% confidence intervals for both models for week 14 include 0.250, making this the thirteenth week that both models did no better or worse than assigning each team a 50% win probability.

In week thirteen of the season, the Elo model had a Brier score of 0.218 and the Cortana model had a Brier score of 0.218.

The 95% confidence intervals for both models for week 13 include 0.250, making this the twelfth week that both models did no better or worse than assigning each team a 50% win probability.

In week twelve of the season, the Elo model had a Brier score of 0.167 and the Cortana model had a Brier score of 0.167.

The 95% confidence intervals for both models for week 12 include 0.250, making this the eleventh week that both models did no better or worse than assigning each team a 50% win probability.

In week eleven of the season, the Elo model had a Brier score of 0.260 and the Cortana model had a Brier score of 0.287. These are the worst performances for the models for the season to date.

Both models performed worse than assigning each team a 50% win probability. The average probability for winning teams in week 11 for the Elo model is 52% and 49% for the Cortana model.

In week ten of the season, the Elo model had a Brier score of 0.189 and the Cortana model had a Brier score of 0.192.

The 95% confidence intervals for both models for week 10 include 0.250, making this the ninth week that both models did no better or worse than assigning each team a 50% win probability. Week 5 has been the only week where the models have performed better than assigning each team a 50% win probability.

In the ninth week of the season, the Elo model had a Brier score of 0.199 and the Cortana model had a Brier score of 0.201.

The 95% confidence intervals for both models for week 9 include 0.250, making this the eighth week that both models did no better or worse than assigning each team a 50% win probability.

In week 8 of the season, the Elo model had a Brier score of 0.217 and the Cortana model had a Brier score of 0.197.

The 95% confidence intervals for both models for week 8 include 0.250, making this the seventh week that both models did no better or worse than assigning each team a 50% win probability. Week 5 has been the only week where the models performed better than assigning each team a 50% win probability. 

In week 7 of the season, both models continued to slide from week 5.

In week six, both models slipped back from their best performance of the season to date in week five.

The Brier scores for the Cincinnati/Carolina tie were calculated by giving one-half of the Brier scores for the win for each team (the Elo model had Cincinnati with a 64% win probability and the Cortana model had Cincinnati with a 72.9% win probability).

Both models had their best performances in week 5 of the NFL season, with the Cortana model performing slightly better than the Elo model.

For week 5, the Cortana model performed about 42% better than assigning all teams a 50% win probability and the Elo model performed about 40% better than assigning all teams a 50% win probability.

In week 4 of the season, the Cortana model performed worse than assigning all teams a 50% win probability.

The 95% confidence interval for the Elo model includes 0.250, making this the fourth week that both models did no better than assigning each team a 50% win probability.

Predictions for both models for the third week of the NFL season were the best for the season. 

The 95% confidence intervals for both models, however, include 0.250, making this the third week that both models did no better than assigning each team a 50% win probability.

In week 2, the Elo model had a Brier score of 0.256, which is worse than assigning each team a 50% win probability. The Cortana model had a Brier score of 0.258, which is also worse than assigning each team a 50% win probability.

In week 1, the Brier scores for both models were slightly better than assigning each team a 50% win probability.

The 95% confidence intervals for the Brier scores for the Elo model and the Cortana model in week 1 include 0.250, making both models no better than assigning each team a 50% win probability.