I trust computer polls, but not with a sub-optimal algorithm

Posted by

Next week the BCS rankings will come out, which will be used to determine who goes to the national championship game. The formula is a combination of 2 human polls – the USA Today Poll and the Harris Poll – and computer rankings.

Being an analyst by nature, I love the science behind the computer polls. You can see how all the computers rank the teams here, and click the links to read their methodology.

I’ve been following Jeff Sagarin of USA Today since middle school, and I would always wake up early on Monday mornings to read how he ranked the teams each week. I always put more trust in his rankings over pollsters, and my intuition wasn’t wrong then.

On his site, he explains how he creates several different algorithms to come up with his rankings…

  • Elo Chess – This is the ranking used for the BCS poll. It only considers wins and losses, not points scored, so Jeff admits it’s sub-optimal
  • Pure Points – This ranking system uses only scoring margin, which is not very politically correct, but the best single indicator of rank.
  • Jeff’s Ranking – The combo of the two systems produces his best predictor of future performance

How do the three systems differ? Greatly. Here’s his Elo Chess top 10:

Boise State #1 and TCU #2? Michigan State #5 and Oregon State #6? Come on. That’s a useless computer ranking. Jeff admits its sub-optimal, but yet he submits it to the BCS and it’s used in the rankings to determine the National Championship Game.

Let’s look at Pure Points, which is the best single prediction method…

Ok, now we’re getting somewhere. I would expect to see Ohio State and Alabama crack the top 10. But it’s still not the best predictor.

Here’s Jeff’s rankings, a combo of the two:

Interesting. Using a combo of pure wins and losses and pure scoring margin, Jeff put together a top-10 that looks different from the polls. Noticeably absent is Ohio State, who is either vastly over-rated or doesn’t have the strength of schedule in the Elo Chess method to match up. It’s also interesting to see how LSU and Nebraska moved up in the combined ranking over either individual ranking. The future will tell how these rankings hold up.

What should we take away from this?

  • It’s absurd for the BCS to use sub-optimal computer polls. Either take the gloves off Jeff Sagarin & his peers, or find a different method.
  • Expect the computer polls to be wrong at first. Since they’re using sub-optimal algorithms, there’s no way they’re going to get it right. And the rankings are more susceptible to change, given that strength of schedule is a primary driver of the model, and many teams have lopsided scheduling
  • Leverage computer polls to identify over-rated and under-rated teams. Michigan State and Oregon State look way over-rated based on wins, losses, and strength of schedule

Leave a Reply

Your email address will not be published. Required fields are marked *