Assumptions

I'm copying and pasting parts of an e-mail exchange that I'm having with Westy about some basketball stats. Mainly, it's me making fun of John Hollinger's power rating statistic, which has somehow become a fixture on ESPN.com. A second issue worth thinking about is how perfectly smart people (like Westy) can fall into common traps. Basically, on the front page of ESPN.com, there was this slick graphic that showed that the Heat were the top team in the NBA sofar, based on John Hollinger's Power Ratings. And of course, I thought that was crap.

My initial e-mail was intended to give a little ribbing to Westy about his love for "advanced" stats like the stuff that John Hollinger uses. Just one sentence and a picture from ESPN.com:

Westy - I guess that I see why you love your advanced stats and hate winning :-)

Note the "winning isn't everything" lead-in. We don't need wins; we have numbers with decimal points.

Also, note that this is before my man D-Will led a 4th quarter comeback, which in turn led to an upset of the Heat by the Jazz in OT.

Of course Westy's response is that Hollinger uses strength of schedule (SOS), with the idea being implied that a statistician like Hollinger wouldn't post anything as ridiculous as a 5-2 team being better than a 7-0 team, unless there was a good reason. To be honest, I had no idea what Hollinger's methodology was, other than it was spitting out shady results. So I took a peek at what was under the hood by looking at the methodology. And of course, I'm appalled.

So, I shoot off an e-mail to Westy. I make some smart ass comment about how going 4-0 against the Nets, T'Wolves and Sixers, while going 1-2 against the Magic, Celts, and Hornets clearly makes you the best team in the NBA. But more importantly, you notice that the Hornets have a higher SOS than the Heat, yet are ranked well below them by Hollinger (despite winning head-to-head). I'm OK with using margin of victory as a component in team evaluation, but you probably shouldn't use it straight up. At this point, I hadn't scrolled far enough down to see the actual equation being used, but I did see that the Hollinger starts off by talking about margin of victory, which suggests that's the major component of this ranking (which is what it looked like at first glance). Part of my comment to Westy:

Seems that Hollinger's first criteria is margin of victory, probably w/out any sort of cap or deeper view beyond the final score. In that sort of scenario, you're rewarded more for winning by 54 points against the T'Wolves and Nets and losing twice by "only" 3 and 8 points (only 2-2, but still +43), than you would be for, say, going 4-0 and winning by 9 points each game (+36). Don't get me wrong. Margin of victory/loss should count, but once you get beyond, say 15 points, you'd have to show me a strong case for why it matters.

Now, I'm sure that Westy didn't look at the Hollinger methodology very closely, and my guess is that he assumed that a statistician that's associated with the "advanced stat" movement wouldn't use a crude tool. At least, certainly not to the extent that I was insinuating. Westy suggests that Hollinger is actually using offensive and defensive efficiency, which would be the best (simple) predictor of team performance. Westy's probably right about how team efficiency stats would be a pretty good predictor, at least compared to the readily available stats. But the assumption is that Hollinger is using something that has built off of those stats.

I had a suspicion that Hollinger's stuff just couldn't be as useful as Westy was suggesting, with the results it was spitting out. So I went back to the explanation of the methodology. And sure enough, we see this:

RATING = (((SOS-0.5)/0.037)*0.67) + (((SOSL10-0.5)/0.037)*0.33) + 100 + (0.67*(MARG+(((ROAD-HOME)*3.5)/(GAMES))) + (0.33*(MARGL10+(((ROAD10-HOME10)*3.5)/(10)))))

Basically, here's what this equation says. Everyone starts with 100. Then, you look at strength of schedule, and measure how much it differs from .500. Then you divide this difference by 0.037 (no explanation for why he uses this number). Do this for the entire season, weighting it 2/3, and do this for the last 10 games, weighting it 1/3 (so that you're placing greater emphasis on the last 10 games). Also, note that he doesn't explain why he chooses the 2/3 and 1/3 weighting (or 10 games for that matter). Finally, you look at the scoring margin, and add an adjustment of 3.5 points for each game played on the road. Again, do this for the entire season, weighting it 2/3, and for the last 10 games, weighting it 1/3. Seems simple enough. But what does this really tell you?

Look at some recent NBA history, readily available on ESPN.com. Since 2002, NBA teams SOS at the end of the season will range from 0.484 to 0.514. The reason that everyone doesn't have a .500 SOS is that you have unbalanced schedules, and the certain conferences/divisions are stronger than others. But if you take the difference from 0.500, and then divide by 0.037, you find that SOS will impact a team's power rating anywhere from -0.378 to +0.432. Okay. So what?

Well, remember that we're starting off at 100 points. SOS impacts you less than half a point either way. So how do you get teams with ratings of 86.814 (the Wizards so far this year) and 116.15 (the Heat, before their loss to the Jazz)? Well, SOS can be a bit skewed right now, but even with a 0.600 SOS, the contribution to the power rating would only be about 2.7. The only other component is scoring margin. Again, if you look at the stats since 1999, you'll see that the lowest/highest scoring margins have been -11.5 and +10.2 points per game. Remember that SOS contributed somewhere between -0.378 to +0.432 points in power rating, we're seeing how scoring margin contributes about 25 times more. This suggests that over 96% of the power rating comes from scoring margin, and less than 4% comes from SOS. Basically Hollinger's power rating is just point differential with a slight tweak based on SOS. Not quite as advanced as something based on offensive and defensive efficiency.

As for the Heat, and their 116.15 power rating? So far this year, their SOS was a very high .595. Divide by .037, and you get 2.57. So, if 100 is the baseline, their SOS contributes 2.57, then their scoring margin contributes the other 13.58. If you place this in context, and look at the actual results, then the interpretation is that the Heat are the best team in the NBA because a) they've smoked the Nets and T'Wolves and Magic, and have lost two relatively close games. Now, if we were in a game where you took all of the points that a team scored in a season, and then subtracted the points that were scored on them, and then awarded a trophy to the team that had the greatest differential, then scoring margin by itself would be a great stat. But, in a game where you have discrete wins and losses, you really should capture the variance of margins in wins and losses. I'm not inclined to look in depth at creating these measures right now (mainly because I'm lazy). But you should be able to tell that once you look under the hood, this Hollinger power rating isn't quite as advanced as the the slick graphics and front-page placement on ESPN.com would have you believe.

In an ideal world, Westy would be safe in his assumption that something that makes the front page of ESPN.com as something from an "advanced stat guy" would be truly useful. Smart people would actually be putting together useful information that extends existing knowledge. Unfortunately, in our world, people have an incentive to sandbag on the truly useful stuff, and instead, we're exposed to the stuff that can fool most of the people most of the time. I'll give Hollinger the benefit of the doubt, and assume that he's got better stuff that he's keeping under wraps, hopefully because it's proprietary for some NBA team that he's consulting. In fact, he even admits that this power ranking needs some caveats. But I don't think that most folks think that it's as crude as I've (hopefully) demonstrated it to be. After all, when you see that the Heat are 116.150, and the Wizards are 88.814, you have all these decimal points that suggest that there's something smart going on under the hood.

-Chairman (aka O.N. Thugs)