American Research Group, Inc.

 

2014 NCAA Men's Division I Basketball Tournament Prediction Accuracy

 

Nate Silver's relaunch of FiveThirtyEight.com includes predictions for the 2014 NCAA Men's Division I basketball tournament. Nate's model provides the odds of each team advancing in the tournament and is updated after each game.

The tables below use Brier scores to determine the accuracy of the probabilistic predictions in Nate's model. Brier scores range from 0 to 1, with 0 meaning the probabilities in the model perfectly match the outcomes of the games and 1 meaning no matches were made (so the closer to 0, the better calibration of the model). If a 50% probability were assigned to each team for each game, the Brier score would be 0.25.

For the 67 games played in the tournament, Nate's model has an overall Brier score of 0.1969. The 95% confidence interval for the overall score is 0.1495 - 0.2441, making this a low performing model (an overall Brier score of 0.0500 or less would make it a high performing model). It should be noted, however, that the Brier score for Nate's model is 0.2559 when calculated by tournament round.

Nate's model performed poorly in the first round of the tournament where the Brier score for the four games of 0.2744 exceeded what the score would have been by assigning each team a 50% probability of winning.

2014 NCAA Men's Division I - Round 1 Play-in
Winner 538 Probability
of Win
Brier
Score
Cal Poly 0.5560 0.1971
Albany 0.5231 0.2274
Iowa 0.4828 0.2675
NC State 0.3631 0.4057
Brier score for 4 games 0.2744

In the second round of play, Nate's model receives a Brier score of 0.1896. Most notable are wins by Harvard, Stanford, North Dakota State, Tennessee, Dayton, Stephen F. Austin, and Mercer with Nate's model showing probabilities of winning for these teams ranging from 42% to 7%.

2014 NCAA Men's Division I - Round 2
Winner 538 Probability
of Win
Brier
Score
Florida 0.9877 0.0002
Arizona 0.9777 0.0005
Wichita State 0.9755 0.0006
Virginia 0.9639 0.0013
Michigan 0.9539 0.0021
Villanova 0.9472 0.0028
Louisville 0.9307 0.0048
Wisconsin 0.9284 0.0051
Kansas 0.9240 0.0058
Michigan State 0.9118 0.0078
Syracuse 0.8830 0.0137
Creighton 0.8831 0.0137
UCLA 0.8708 0.0148
Iowa State 0.8124 0.0352
San Diego State 0.7455 0.0648
Kentucky 0.7393 0.0680
Pittsburgh 0.7243 0.0760
Baylor 0.7032 0.0881
North Carolina 0.6796 0.1026
Connecticut 0.6730 0.1069
Oregon 0.6473 0.1244
Saint Louis 0.5805 0.1760
Memphis 0.5500 0.2025
Texas 0.5012 0.2488
Gonzaga 0.4799 0.2705
Harvard 0.4198 0.3366
Stanford 0.3625 0.4065
North Dakota State 0.3621 0.4069
Tennessee 0.3600 0.4096
Dayton 0.2469 0.5671
Stephen F. Austin 0.2359 0.5839
Mercer 0.0708 0.8633
Brier score for 32 games 0.1896

In the third round of the tournament, the Brier score for Nate's model is 0.1899. Most notable in this round are wins by Baylor, Kentucky, Connecticut, Dayton, and Stanford with Nate's model showing probabilities of winning for these teams ranging from 46% to 21%.

2014 NCAA Men's Division I - Round 3
Winner 538 Probability
of Win
Brier
Score
UCLA 0.8782 0.0148
Louisville 0.8223 0.0316
Tennessee 0.8106 0.0359
Michigan 0.7869 0.0454
Michigan State 0.7723 0.0518
Wisconsin 0.7720 0.0520
Florida 0.7620 0.0566
Arizona 0.7266 0.0748
Virginia 0.7099 0.0841
San Diego State 0.6458 0.1254
Iowa State 0.5572 0.1960
Baylor 0.4560 0.2960
Kentucky 0.4214 0.3348
Connecticut 0.3510 0.4212
Dayton 0.2266 0.5981
Stanford 0.2127 0.6198
Brier score for 16 games 0.1899

The Brier score Nate's model in the fourth round is 0.2166. This is about 13% better than assigning all teams a 50% probability of winning. Kentucky was an upset based on Nate's model.

2014 NCAA Men's Division I - Round 4
Winner 538 Probability
of Win
Brier
Score
Arizona 0.7263 0.0749
Florida 0.7164 0.0804
Wisconsin 0.5908 0.1674
Michigan 0.5254 0.2252
Connecticut 0.5159 0.2343
Michigan State 0.5072 0.2428
Dayton 0.5008 0.2492
Kentucky 0.3230 0.4584
Brier score for 8 games 0.2166

The Brier score for Nate's model for the fifth round exceeds what the Brier score would have been by assigning each team a 50% probability of winning. Wisconsin and Connecticut were upsets based on Nate's model.

2014 NCAA Men's Division I - Round 5
Winner 538 Probability
of Win
Brier
Score
Florida 0.8306 0.0287
Kentucky 0.5485 0.2038
Wisconsin 0.3681 0.3993
Connecticut 0.3658 0.4022
Brier score for 4 games 0.2585

In round 6, Nate's model missed both winners by a large margin.

2014 NCAA Men's Division I - Round 6
Winner 538 Probability
of Win
Brier
Score
Kentucky 0.4183 0.3383
Connecticut 0.3007 0.4890
Brier score for 2 games 0.4137

In the final round, Nate's model picked the winning team, but only after recalculating the probabilities for the final game.

2014 NCAA Men's Division I - Round 7
Winner 538 Probability
of Win
Brier
Score
Connecticut 0.5013 0.2487
Brier score for 67 games 0.1969

 

How well is Nate's model calibrated?

Nate's model performed poorly in rounds 1, 5, and 6, Nate's model had the final game virtually a toss-up, and the model did not improve as the tournament progressed. The overall Brier score of 0.1969 confirms that this is a low performing model.

ARG Home