S&P+ rankings: Overperformance or overcorrection?

[satyanash 14276]

10 minutes ago, sushihorn said:

For fumbles he uses a projected recovery rate rather than actual recoveries and attributes the difference to luck.  It ignores context but if all you have is aggregate data it makes sense within those limitations.  Of course,= in reality there is a very big difference between a fumbled snap and a ball ripped out with a crowd of defenders waiting to pounce.  

Bill's projected rate is based on forced fumbles and passes defensed. So in your above example, points from the fumbled snap would be attributed to luck, while the ripped-out ball counts as a forced fumble and is included based on the assumption that 50% of fumbles are recovered by the opposing team for a turnover.

[Magus Ossis 2773]

If Connelly has not understood or modeled something, it is "luck." The sheer audacity/ stupidity of this worldview is astounding. I am not a betting man, but taking the over on his estimation of Texas wins in 2019 seems as close to free money as exists.

Doctor Connelly: I gave him penicillin and plenty of intravenous hydration. Just bad luck that he died.

Narrator: He had NSC lung cancer that was never resected.

[Huckleberry 32685]

1 minute ago, satyanash said:

Bill's projected rate is based on forced fumbles and passes defensed. So in your above example, points from the fumbled snap would be attributed to luck, while the ripped-out ball counts as a forced fumble and is included based on the assumption that 50% of fumbles are recovered by the opposing team for a turnover.

A fumbled snap isn't a forced fumble in his numbers?

[satyanash 14276]

1 minute ago, Huckleberry said:

A fumbled snap isn't a forced fumble in his numbers?

If it is, I've been making a lot of wrong assumptions.

[Huckleberry 32685]

I've always been under the impression that he merely looks at opponents' fumbles. I'm not aware of any reliable source that differentiates between forced fumbles and your opponent just dropping the ball, although just because I'm not aware of it doesn't mean that it doesn't exist.

https://www.footballstudyhall.com/2015/1/30/7947287/college-football-turnover-luck-2014-tcu-oklahoma

That article indicates it's simply fumble recovery percentage.

[Valmy77 13092]

We had 16 fumbles and lost 5 or them. I don't know how many were forced though.

[TheAuditor 1030]

I used S&P+ to do a bowl pick'm league (against the line) with about 200 players to see how it did across a large number of teams and large number of games.  I finished 183.  As an observer of past outcomes, its interesting.  As a predictor,  I think Vegas has this sorted.  

[satyanash 14276]

Here's some data on how S&P+ performed against the spread in 2018: https://docs.google.com/spreadsheets/d/e/2PACX-1vTNXgxlcihtmzIbzHDsQH5CXI6aSXfsZzWB7E8IC0sf4CaMsgP_p4DRSwx6TtoektFRCL3wO5m64JLB/pubhtml#

Started out really well, but regressed as the season went on to end up at 51.3% - which seems like the opposite of what you'd expect a predictive model to do as it gains more data. Over the last few years S&P+ has consistently outperformed the spread over 50% of the time, but not by a huge margin.

2015 - 51.85%

2016 - 50.07%

2017 - 52.52% (50.58% adjusted)

Edited February 11, 2019 by satyanash

[Valmy77 13092]

How does that compare to other statistical models? Maybe we should be concerned with better ones and less with the S&P+ if it is that bad against the spread?

[Magus Ossis 2773]

It would be interesting to see how S&P+ does against the spread versus some other complex model. I suggest merely picking home underdogs to cover and seeing whether it approximates or surpasses any of his season results.

[Huckleberry 32685]

That's not "bad" against the spread, but it's still losing money. Let's put it this way, if Bill Connelly had developed a system that could beat the spread at a 55% rate consistently he would be an absolute fool to publish it.

(52.4% is required to beat the standard -110 juice)

In the interest of full disclosure my system beat the spread on high confidence games at over a 55% clip for two seasons about a decade ago. But for the last few seasons it is well south and right at the 50% mark. Being able to consistently hit that number would be incredible. We are talking really fine margins here when it comes to betting against the spread. It's not like 90% or even 70% is the target mark for excellence. There is a range from 52.38% where you lose money to 55% where you're filthy rich.

Edited February 11, 2019 by Huckleberry

[satyanash 14276]

5 minutes ago, Huckleberry said:

I've always been under the impression that he merely looks at opponents' fumbles. I'm not aware of any reliable source that differentiates between forced fumbles and your opponent just dropping the ball, although just because I'm not aware of it doesn't mean that it doesn't exist.

https://www.footballstudyhall.com/2015/1/30/7947287/college-football-turnover-luck-2014-tcu-oklahoma

That article indicates it's simply fumble recovery percentage.

Bill uses forced fumbles as part of his calculation for "Havoc Rate", so he definitely keeps track of it: https://www.footballoutsiders.com/stats/ncaadef

Quote

Havoc rate is calculated by tallying the total number of tackles for loss, passes defensed (interceptions and breakups), and forced fumbles and dividing it by total plays.

Bill also mentions using forced fumbles and sack rates to calculate a team's expected turnovers in this article.

[Huckleberry 32685]

2 minutes ago, satyanash said:

Bill uses forced fumbles as part of his calculation for "Havoc Rate", so he definitely keeps track of it: https://www.footballoutsiders.com/stats/ncaadef

Bill also mentions using forced fumbles and sack rates to calculate a team's expected turnovers in this article.

Right, but my belief is that forced fumbles in those sentences really only means opponents' fumbles. 

Edited February 11, 2019 by Huckleberry

[texifornia 56434]

40 minutes ago, Pato del Muerto said:

If you give him LSU and ou as losses, I wonder which teams he picks for the other 3 losses.

Okie Lite and TCU are both rated above us, plus the cumulative likelihood that we drop a game to one of the lesser teams.

[HtownHorn 14446]

His model also makes no notation of when the TO happened. Like when Texas recovered the fumble when OU was throwing the ball around on the last play of the game. That TO was inconsequential to the outcome of the game, but his model still used it as it was a major factor.

[Pato del Muerto 52322]

31 minutes ago, Huckleberry said:

That's not "bad" against the spread, but it's still losing money. Let's put it this way, if Bill Connelly had developed a system that could beat the spread at a 55% rate consistently he would be an absolute fool to publish it.

(52.4% is required to beat the standard -110 juice)

In the interest of full disclosure my system beat the spread on high confidence games at over a 55% clip for two seasons about a decade ago. But for the last few seasons it is well south and right at the 50% mark. Being able to consistently hit that number would be incredible. We are talking really fine margins here when it comes to betting against the spread. It's not like 90% or even 70% is the target mark for excellence. There is a range from 52.38% where you lose money to 55% where you're filthy rich.

Do you know of any % from a random choice study?  Since the spread is designed, more or less to even out the money, it’s kind of a rough proxy for a 50/50 choice. 

So the floor for random choices can’t be close to zero, can it?  It’s got to be in the 20s or 30s, at least for spreads inside of around 15 points. 

[Huckleberry 32685]

Random would indeed just be 50%. So I've always looked at it like 50% is random selection, 55% is beyond excellent, and halfway between happens to be roughly the breakeven point.

[satyanash 14276]

 

[Huckleberry 32685]

That's the other humorous thing, how defensive he gets in conversations. Saying that his system completely nailed the 2017 Texas team is silly. His preseason projection missed significantly not only compared to end-of-season polls and rankings, but also to his own system at the end of the year. The only place it was "nailed" was his second order wins which is just an ex post facto look at how likely the team who won was likely to win based on the stats.

But of course that is basically meaningless. I could create a system much more accurate at such an analysis, takes about two seconds. Which team had more points at the end of the game? This system is 100%. His entire argument, which is completely valid, is that looking deeper than the final score is important when predicting future performance. But that has absolutely nothing to do with analyzing past performance. 

His system did not completely nail Herman's 2016 and 2017 teams, that's simply false.

[DanRydell 799]

1 hour ago, Pato del Muerto said:

It’s weird that he only talks about final scores, presumably because that’s what his model looks at. 

I don't think his model actually knows what the final score is, or even if a team won or lost the game. The only relevance score has in his model is determining whether or not a game is in garbage time.

[DanRydell 799]

1 hour ago, Magus Ossis said:

It would be interesting to see how S&P+ does against the spread versus some other complex model. I suggest merely picking home underdogs to cover and seeing whether it approximates or surpasses any of his season results.

IIRC, it's not actually modeled to predict against the spread but rather to predict win probabilities. I have no idea how well it performs at that (do teams predicted to win a game 70% of the time actually win roughly 70% of the time?) or how it compares to any other model at doing so.

[Huckleberry 32685]

Just now, DanRydell said:

IIRC, it's not actually modeled to predict against the spread but rather to predict win probabilities. I have no idea how well it performs at that (do teams predicted to win a game 70% of the time actually win roughly 70% of the time?) or how it compares to any other model at doing so.

If that's the case then he should go with that, because I have found that Vegas moneylines are much softer than point spreads.

[Magus Ossis 2773]

I didn't see numbers out there for college football, but for the NFL, the favorite wins over 68% of the time. With the greater disparity in talent among college teams (v pro), one would expect a higher fraction of games that are easier to predict. If I wanted a model to tell me with circa 70% accuracy who would win, I could just use the line. The devil is in reliably predicting wins significantly better than the line does. Connelly gets right that he needs to dig deeper than final score. He gets a great deal wrong after that.

[golfclap 6903]

Connelly is a redass Mizzou grad that hates Texas and people are surprised when his "predictive models" always have Texas/Herman as a constant outlier? 

fuck that guy. 

[Magus Ossis 2773]

EDIT: I meant to quote my own question about accuracy ATS.

The answer I found was NFL, not NCAA, but the home dog coverage rate was 51.7%

Cover.com

So the S&P+ can achieve win percentage predictions similar to always picking the favorite and similar results ATS as always picking home dogs. Based on accuracy of this magnitude, he attributes to “luck” what his model gets wrong. Clearly his model does somewhat correlate to real life and therefore has some predictive power. He far overestimates the science in what he does, however.

Edited February 11, 2019 by Magus Ossis

[Tex Pete 20136]

22 minutes ago, golfclap said:

Connelly is a redass Mizzou grad that hates Texas and people are surprised when his "predictive models" always have Texas/Herman as a constant outlier? 

fuck that guy. 

Has anyone ever seen Connelly and DanRydell in the same room together?

[Fozzz 152]

23 minutes ago, golfclap said:

Connelly is a redass Mizzou grad that hates Texas and people are surprised when his "predictive models" always have Texas/Herman as a constant outlier? 

fuck that guy. 

This is so aggy.

[Huckleberry 32685]

I don't believe Connelly has any particular axe to grind against Texas, it's just that he feuds with Texas fans more than anyone else right now because of the disconnect between his system and Texas's big wins last season.

The only biases I see come through in his writings are being defensive about his work (understandable and not even an issue) and being exuberant about anything that reflects positively on his alma mater Missouri. The latter bias is the only one that would seem like a problem in terms of objectivity but it's largely unavoidable and I haven't seen any evidence it causes any issues for his work's integrity.

[Valmy77 13092]

Yeah I just do not understand his model, I think it highly unlikely he set out to make one that made Texas look bad. We do that on our own in the 4th quarter.

I don't get how returning Cosmi, CJ, Duvernay, Keontay Ingram, Sam Ehlinger, Caden Sterns, BJ Foster, Brandon Jones, Malcolm Roach, Jeffrey McCulloch and so forth is the 8th ranked returning group in the Big 12 and the 51st ranked returning group in the nation. If your statistics are producing that result then I question your model. 

[vin2basketball 10]

1 hour ago, Valmy77 said:

Yeah I just do not understand his model, I think it highly unlikely he set out to make one that made Texas look bad. We do that on our own in the 4th quarter.

I don't get how returning Cosmi, CJ, Duvernay, Keontay Ingram, Sam Ehlinger, Caden Sterns, BJ Foster, Brandon Jones, Malcolm Roach, Jeffrey McCulloch and so forth is the 8th ranked returning group in the Big 12 and the 51st ranked returning group in the nation. If your statistics are producing that result then I question your model. 

LJH was a 1,000 yard receiver, we lost two senior corners that tackled a lot and our entire DL + Gary Johnson, and Tre Watson was our leading rusher thanks to his late in the year surge, though Keaontay Ingram is arguably more talented.

That's not to say that if McCoy is cleared, then the freshmen plus a year of growth from Duve can't replace LJH, that Ingram can't be a feature back, and that our stable of safeties and young corners can't ride their talent to comparable results. But statistically, that's not a common thing to bet on - only the very best teams (Clemson, Bama, etc) tend to match performance YOY despite a significant youth movement.   

Edited February 11, 2019 by vin2basketball

[Valmy77 13092]

Just now, vin2basketball said:

LJH was a 1,000 yard receiver, we lost two senior corners that tackled a lot and our entire DL + Gary Johnson, and Tre Watson was our leading rusher thanks to his late in the year surge, though Keaontay Ingram is arguably more talented.

No I get that we lost some dudes...but didn't everybody?

[vin2basketball 10]

1 minute ago, Valmy77 said:

No I get that we lost some dudes...but didn't everybody?

His system overweights DB tackles in a particularly weird way for some reason - so the loss of Boyd + Davis looms large in his system

Edited February 11, 2019 by vin2basketball

[Machinator 32097]

4 minutes ago, Valmy77 said:

No I get that we lost some dudes...but didn't everybody?

We lost 8 out of 11 starters on defense.

[texifornia 56434]

2 hours ago, golfclap said:

Connelly is a redass Mizzou grad that hates Texas and people are surprised when his "predictive models" always have Texas/Herman as a constant outlier?  

fuck that guy. 

(He also grew up an OU fan before going to Mizzou)

Honestly, I think he tries to be impartial, but Texas fans drive him off the edge faster than any other group for obvious reasons.

Edited February 11, 2019 by texifornia

[Pato del Muerto 52322]

1 hour ago, Huckleberry said:

I don't believe Connelly has any particular axe to grind against Texas, it's just that he feuds with Texas fans more than anyone else right now because of the disconnect between his system and Texas's big wins last season.

The only biases I see come through in his writings are being defensive about his work (understandable and not even an issue) and being exuberant about anything that reflects positively on his alma mater Missouri. The latter bias is the only one that would seem like a problem in terms of objectivity but it's largely unavoidable and I haven't seen any evidence it causes any issues for his work's integrity.

Well his retort seems to be that his model is not wrong, herman’s team just did better than it should have two out of 4 years and he attributes that to “luck.”  

How well does his luck measureable equate to win/loss performance deviations for teams?  

There must be many instances of significant amounts of luck for a team, good or bad, in games throughout the year. 

If a team like Texas, or any other that the numbers beat out, seem to be good and consistent at getting lucky, maybe there’s more to it than luck and some data parsing may be in order. 

[Fozzz 152]

2 minutes ago, Pato del Muerto said:

If a team like Texas, or any other that the numbers beat out, seem to be good and consistent at getting lucky, maybe there’s more to it than luck and some data parsing may be in order. 

Herman is "consistently" lucky if two seasons out of four is consistent.  In other words, 50% of the time Herman's magic works 100% of the time.

[Huckleberry 32685]

That seems to be the exact same percentage of the time that Connelly's magic works on Herman's team. Somehow for you 50% doesn't mean that Herman defies Connelly's system but 50% does mean that Connelly nails Herman's teams.

[Valmy77 13092]

13 minutes ago, Machinator said:

We lost 8 out of 11 starters on defense.

It seems impossible to get a straight answer to my question.

Yes I get that we are having lots of guys leave for the NFL and graduate. I presume most teams are not starting an entire roster of Freshman and Sophomores though. Surely this does not make us among the worst in the Big 12 as far as returning players. I mean the entire reason we are #4 in the Big 12 by his numbers seems to be inferior returning players to OU, OSU, Baylor, Tech, TCU, West Virginia, and Iowa State. So the whole conference outside of Texas and the Kansas schools are bringing everybody back and therefore Texas should lose five games? Because I don't see how his other two columns would lead him to this conclusion.

Edited February 11, 2019 by Valmy77

[Fozzz 152]

6 minutes ago, Huckleberry said:

That seems to be the exact same percentage of the time that Connelly's magic works on Herman's team. Somehow for you 50% doesn't mean that Herman defies Connelly's system but 50% does mean that Connelly nails Herman's teams.

If Herman possessed a magic ability skill to consistently outperform S&P+, you would expect it to show up more or less consistently, and not half the time.  It would naturally fluctuate in degree, as all skills do, but you would expect some season-to-season consistency.  I mean, if the magic skill comes and goes at random then it's not much of a skill is it?  Maybe the magic skill is cyclical and we will need to wait until 2021 for it to make its presence known again.  Maybe the right prayers will hasten its arrival, I'm not sure.

[TheFlagship 432]

I’m a big fan of using data in sports for use in predictive decision making, but college football is too random to build an accurate model.

In most cases, you’re talking about 19-20 year old kids that are still learning and still developing on a weekly, if not daily basis. What occurred in the past is going to be less predictive of the future than a model for any pro league. The rosters are also giant so it’s difficult to predict and project playing time and samples are so small, it can be challenging to quantify actual player skill. 

I admire his intent in trying to quantify results and I think it does portray what happened fairly well, but college football is often random as fuck as we know, and I can’t buy into any predictive value being derived from S&P. I’ll leave that for aggy suckers

[Huckleberry 32685]

4 minutes ago, Fozzz said:

If Herman possessed a magic ability skill to consistently outperform S&P+, you would expect it to show up more or less consistently, and not half the time.  It would naturally fluctuate in degree, as all skills do, but you would expect some season-to-season consistency.  I mean, if the magic skill comes and goes at random then it's not much of a skill is it?  Maybe the magic skill is cyclical and we will need to wait until 2021 for it to make its presence known again.  Maybe the right prayers will hasten its arrival, I'm not sure.

Every argument you make in that direction could be applied to Connelly's system when it is applied to a team which clearly focuses on drive efficiency versus explosive plays. Connelly has already modified his system once when he decided that he was placing too much emphasis on explosiveness when in reality it is merely a component of drive efficiency. Given his constant tinkering, which isn't a problem, why do you seem so convinced that his current model/system is the final and most accurate one?

[sushihorn 3233]

11 minutes ago, Fozzz said:

If Herman possessed a magic ability skill to consistently outperform S&P+, you would expect it to show up more or less consistently, and not half the time.  It would naturally fluctuate in degree, as all skills do, but you would expect some season-to-season consistency.  I mean, if the magic skill comes and goes at random then it's not much of a skill is it?  Maybe the magic skill is cyclical and we will need to wait until 2021 for it to make its presence known again.  Maybe the right prayers will hasten its arrival, I'm not sure.

Herman has outperformed his S&P expected wins every year.  It just happened to be by less than a full game on 2 occasions (0.7 and 0.5 IIRC).  The other 2 times it was a much larger deviation.  IOW he's been wrong about Herman and in the same direction for all 4 years as a HC.  He only gets it REALLY BADLY WRONG 50% of the time.

This year should be a good test case.  7 wins sounds very low to me.  If Texas doesn't have a massive wave of injuries and only wins that many I'd say Connally will be partially vindicated.  If Texas wins 9+ it's further proof that his models are really bad for Tom Herman teams.

Edited February 11, 2019 by sushihorn

[texifornia 56434]

 

[Fozzz 152]

Just now, Huckleberry said:

Every argument you make in that direction could be applied to Connelly's system when it is applied to a team which clearly focuses on drive efficiency versus explosive plays. Connelly has already modified his system once when he decided that he was placing too much emphasis on explosiveness when in reality it is merely a component of drive efficiency. Given his constant tinkering, which isn't a problem, why do you seem so convinced that his current model/system is the final and most accurate one?

I never said it was.  I also don't think the latest iteration of S&P+ puts much emphasis at all on explosive plays, as evidenced by our offensive S&P+ being about the same as our marginal success rate in spite of an absolutely putrid IsoPP. 

As I've said previously, you would think Herman's offensive system is built for dominating S&P+ given that it emphasizes efficiency (stringing together successive successful plays) rather than looking to create explosive plays while risking more negative plays.  Herman's offensive system is the analytically "smart" thing to do given how noisy explosive plays are.  

[Huckleberry 32685]

2 minutes ago, texifornia said:

 

So it's consistently coaches whose offenses rank better at drive efficiency (yards per possession) and turnovers than they do at explosiveness (yards per play). That makes perfect sense.

[Huckleberry 32685]

3 minutes ago, Fozzz said:

I never said it was.  I also don't think the latest iteration of S&P+ puts much emphasis at all on explosive plays, as evidenced by our offensive S&P+ being about the same as our marginal success rate in spite of an absolutely putrid IsoPP. 

As I've said previously, you would think Herman's offensive system is built for dominating S&P+ given that it emphasizes efficiency (stringing together successive successful plays) rather than looking to create explosive plays while risking more negative plays.  Herman's offensive system is the analytically "smart" thing to do given how noisy explosive plays are.  

It's a give-and-take. Explosiveness helps when you screwed up first down and are facing second-and-long. But if you're still maintaining a high drive efficiency without explosiveness than that would imply that you do a good job of not screwing up first down as often as other teams.

[Fozzz 152]

4 minutes ago, sushihorn said:

Herman has outperformed his S&P expected wins every year.  It just happened to be by less than a full game on 2 occasions (0.7 and 0.5 IIRC).  The other 2 times it was a much larger deviation.  IOW he's been wrong about Herman and in the same direction for all 4 years as a HC.  He only gets it REALLY BADLY WRONG 50% of the time.

This year should be a good test case.  7 wins sounds very low to me.  If Texas doesn't have a massive wave of injuries and only wins that many I'd say Connally will be partially vindicated.  If Texas wins 9+ it's further proof that his models are really bad for Tom Herman teams.

2016 and 2017 were both 0.5 deviation from 2nd order wins, which is a small deviation overall as many teams have a greater deviation between actual and 2nd order wins than 0.5.  For instance, there were 45+ teams last season with a difference of 1.0 or greater.  There was about as many teams with a 1.0 or greater difference than there were teams with 0.5 or less.  Do you only consider it getting right if the difference is 0.0?  

[sushihorn 3233]

1 minute ago, Fozzz said:

2016 and 2017 were both 0.5 deviation from 2nd order wins, which is a small deviation overall as many teams have a greater deviation between actual and 2nd order wins than 0.5.  For instance, there were 45+ teams last season with a difference of 1.0 or greater.  There was about as many teams with a 1.0 or greater difference than there were teams with 0.5 or less.  Do you only consider it getting right if the difference is 0.0?  

I consider it important as part of a pattern of always getting it wrong in the same direction.  If half the deviations were positive then I'd excuse the smaller ones as mere randomness.  Combine that with his real outlier of 7 wins for 2019 and it's a pretty strong pattern too.

[Fozzz 152]

One way you could potentially play "above" your S&P+ would be to play down to your competition, like our shitty effort against Tulsa this season.  Not something I would consider a real asset or skill for a HC though.  

[Magus Ossis 2773]

His system necessarily uses proxies because of the components he incorporates. Blake Gideon was a starter on our defense, and in a hypothetical universe in which he was replaced by Sterns, the S&P+ would have punished us for it. It would also have rewarded us for recruiting Gilbert, and for returning him to the line-up (versus penalty had we picked up a stud grad transfer). Generally speaking, returning production is a proxy for producing needed yards/points (offense) or stops (defense) in the future. He chooses proxies which have shown positive correlation in pooled data, and they likely show correlation because of some true connection. When proxies do not align as well with the true values as one would desire, the answer is not to criticize the truth for failing to follow the model. There are teams that tend to underperform S&P+ predictions in various ways, teams that cluster nicely in the hump of the curve, and teams that tend to over perform it. Texas will win 2+ (4 would be unsurprising, frankly) more games than predicted next year, and Connelly will call it a fluke or luck. Eventually, as the young talent at Texas becomes mature talent, our on-field results will result in a higher S&P+ and hence smaller delta between predicted and final outcomes. Connelly will then note that even the "Herman luck" ran out, never admitting that his model rather finally caught up closer to reality.