Yes, Duke is quite bad. I can’t believe Kenpom still ranks them in the low 30’s. Torvik is more realistic by ranking them in the 60’s.
1 Like
You expect some variance, but that’s quite a gap between two widely-used systems we expect to be better than the polls.
I don’t know, I feel like it’s actually a similar story to recruiting in terms of where the rank ordering really means something. Once you’re out of the top 30 but still in the top 100-150, there’s enough game-to-game variation that it’s hard to tell teams apart statistically.
1 Like
Could be. Dunno. Arguably though you need a data-driven ranking system to be meaningful at the very least for roughly the top 40, considering the number of autobids for the tournament. But definitely a lot deeper if the concept is to fulfill its promise.
While I see what you’re saying and think there’s some truth to that…you would never hear KenPom himself say that. The tournament takes 68 teams - and 36 of them are at-large. You better be able to sort out the teams at least through about 50-60.
I’m not taking a shot of KenPom (I think he’s by far the best at what he does), I’m more making a methodological point that it’s kind of an impossible task. What KenPom (and I assume Torvik, but I’m not as familiar with his site) reports is how well a team has played, not how good they are or how well they’d do in any one-off game, which is really what you’d want to rank teams on for the tournament. That means that a team’s “true” adjusted efficiency (or whatever metric you like) is in an error band around what KenPom reports, and the more game-to-game variance in the sample of games played means the error band is wider. So there are bound to swaths of teams, especially as you move down the rankings, that are indistinguishable because those error bands overlap. Obviously the fair thing to do is to rank teams based on how they have actually played, not some estimate of how they might play. I was just pointing out the rankings aren’t as precise or objective as some people make them out to be.
KenPom isn’t designed to pick teams for the NCAA Tournament, it is to provide predictive metrics between teams. I don’t really think there’s any drop off as you go further down the rankings, either. The comparison to recruiting rankings also doesn’t make complete sense to me because that’s more a product of actual limitations (visibility, evaluation, etc.) whereas rankings have all of the same data.
If anything, the actual numeric differences in Eff Margin are probably much smaller between the actual ranking spots further down which could create some perceived differences but that’s also just why you should look more closely at the numbers.
I was just using KenPom since everyone is most familiar with his methodology. And yeah, I agree about recruiting rankings, the source of uncertainty is completely different.
Yeah, that’s kind of my point. The point estimates are closer together and the margins of error (although KenPom doesn’t report them) are sure to be larger because the teams are just more volatile game-to-game. So the further down the rankings you go, the less certain you can be that a higher ranking actually means a better team.
Why are teams more volatile game-to-game further down the rankings? KenPom’s site has the actual Adj Eff numbers and also “luck” which is essentially volatility in performance versus results.
I just don’t follow why KenPom is somehow less effective in different ranges of the rankings.
I was making an assumption there that good teams mostly play well, bad teams mostly play badly, and middle of the road teams tend to bounce between good and bad games. I don’t actually have anything to back that up. I don’t think luck is at all the same as what I’m talking about. Luck is about random swings in teams’ past results, not the statistical uncertainty of making inferences about future results based on a limited sample of past results.
I don’t know if what I’m about to suggest is accurate or merely an impression, but it’s the presence of Kenpom and other data-driven methodologies that have led the NCAA to adopt its own, so my impression is that whatever the stated purpose, there’s a good degree of venn-diagram-like overlap in what these things do and to what end.
The overlap should be that predictive metrics better help understand how good a win or loss is. You need something that decides how good one team is relative to another to then evaluate a resume on. A team can be top 25 good but have a bad resume (ex penn st this year). You would use the predictive metrics to justify that wisconsin losing on the road at penn st isn’t a terrible loss. But by the same tool you can say that penn st’s resume isn’t good enough to get into the tournament. You still evaluate seeding teams on the same results based system you had before, but now you don’t over value a road loss to a decent team just because that team has a bad overall record.
1 Like
The Committee uses NET in a drastically different manner than people use Kenpom. They’re not just seeding 1-36 straight up with NET. They use NET to compare team resumes to each other.
1 Like
I really can’t say for sure, but I think it’s important to think about and not just a sports thing. People are trying to measure things and that’s good, and so much better than before we had the ability to work with big data, but sometimes if you pick at data-driven conclusions out there in the working world you discover that they are really just hottakes because somebody made a dump assumption about data.
Thanks. Where can I learn more? Read anything good about this?
Just look over the team sheets they provide and you can see how they’re using NET. I can never find them, but @umhoops usually has them handy.
Torvik has team sheets. Not sure if they are the same exactly as what they use though.
My intuition tells me that you are correct that there is more volatility as you move down the rankings. There are 350 division 1 schools. We can group them in 7 buckets of 50 teams. What feels like the safer bet? 1 vs 50 or 301 versus 350?
Edit: actually the more I think about it there is probably more consistency at the extremes and the volatility is in the middle. The worst basketball team out of 350 is probably as extremely bad as the #1 team is probably extremely good.