ESPN's QBR Stat Falls Short Of Revolutionary

By Scott Kacsmar
August 17, 2011 5:54 pm
Print
1,353 Views 0 Comments
From the network that brought you “The Decision,” ESPN has unveiled a new statistic to rate quarterbacks that they believe will revolutionize the evaluation of quarterbacks.
 
It is called Total Quarterback Rating, or QBR for short, and it will focus on every play a quarterback makes in a game except for handing the ball off and audibles. Rather than just focusing on passes thrown like the conventional NFL passer rating formula, QBR will also look at sacks, fumbles, runs, and penalties.
 
This is a large-scale project that takes into account statistics and video analysis to create the rating. While many of the concepts are not new, they have never been combined in such a way to attempt creating the perfect stat. The worldwide leader seems pretty confident this will replace the old passer rating system.
 
However, don’t start digging that grave for passer rating just yet.
 
Methodology
 
Back in the 1980’s, Pete Palmer, Bob Carroll, and John Thorn created a book called “The Hidden Game of Football”. It was football’s first answer to Bill James’ sabermetric-approach. Included in the book was the concept of “expected points” that the offense can gain on each play. The closer you are to your opponent’s end zone, the higher your expected points will be. You can also have a negative number for expected points if you are backed up in your own end and close to punting.
 
Building on that theory, ESPN used ten years of play-by-play data (which sounds awfully similar to Brian Burke’s work at Advanced NFL Stats) to create an expected point value for each play based on the down, distance, field position, time and score of the game.
 
The quarterback is graded on the change in expected points before and after each play. That number is then divided among the quarterback and his offensive teammates based on video review of what happened on the play. There is no real mathematical explanation of how the credit/blame is split between the quarterback, his linemen, receivers and backs. Just know that the sum of the offensive players will add up to the total of expected points for that play.
 
There is also a clutch index which will give extra weight to game situations that are more critical to win probability (late in the fourth quarter) while lowering the weight of less critical situations (garbage time drives). The numbers are averaged together on a scale that goes from 0 (worst) to 100 (best) with 50 being average. QBR is available for each quarterback for the 2008-2010 seasons only.
 
Essentially the quarterback is being graded on what he does each play, and in what situation it happened.
 
Usually one would post the formula for their system, but that is not going to happen here. ESPN has thousands of lines of code devoted to QBR, and we just aren’t going to see it.

The Pros

It is a good thing to see a company like ESPN understands that the current passer rating system does not do a great job of representing a quarterback’s performance in a given game. Other statistical sites like Cold, Hard Football Facts and Football Outsiders have worked on advanced stats for years, so ESPN joining along could help the football stat community. At the very least, they will offer up another stat for comparison.
 
Without getting into specifics, so far it does appear to do a decent job of figuring out which quarterbacks are the best, the worst and average over the last three seasons.
 
Many of the concepts involved in QBR are very strong and valid points in analyzing the quarterback position:
  • A player is best judged by his efficiency, not his volume stats
  • Gives more credit to quarterbacks that thrive in situations like third and long and the red zone
  • Does not punish quarterbacks that throw Hail Mary interceptions in desperate situations
  • Lessens the impact of garbage time stat-padding
  • Yards after catch (YAC) has more to do with the receivers than it does the quarterback; same with dropped passes
  • Gives more credit to quarterbacks that throw deep, accurate passes rather than short, easy screens that look like glorified runs
  • Quarterbacks will finally be punished for their sacks and fumbles, while rewarded for the ability to scramble and score rushing touchdowns
  • A scale that goes 0-100 is a big improvement over the capped 0-158.3 scale, though there should still be a way to go into negative numbers
The notion to include a clutch index is good, because there are situations in a game that carry more weight and pressure than others, and that type of success should be rewarded. However, actually applying this mathematically in a way that’s fair is practically impossible (more on that later).
 
Perhaps the most important stat on QBR was saved for the end of ESPN’s Friday night special. The team with the higher QBR has won 85.7% of the games played since 2008. That is an incredible winning percentage, higher than winning the turnover differential and passer rating battle. This could give much credence to the thought that the team whose quarterback plays better should win the game.

The Cons

Uhh…Exactly How Does This Work?
What could be the biggest hurdle for ESPN in establishing QBR is the fact that no one is going to know how to calculate it, or what exactly a quarterback would have to do to improve his rating.
 
You might say, “But you don’t know how to calculate passer rating.” Maybe not off the top of my head, but one can quickly do a Google search and find the formula, or better yet, find a site that can easily calculate it for me. You can calculate passer rating on your own for any single play (not very useful) or any group of plays you find.
 
That’s never going to happen with QBR, folks. Even if ESPN releases a mega-programmed calculator on their site, who would want to sit there and type in every input necessary (which includes numbers that can only be recorded from video analysis) for every single play the quarterback made in a game so that it spits out that magical number? It will never happen.
 
Dividing Credit Sounds Too Subjective
What are the chances JaMarcus Russell received the same leeway as Aaron Rodgers on an incomplete pass that may not have been the quarterback’s fault? The division of credit is an ill-explained system that reeks of subjectivity, which means a lack of consistency. This is bad news for the credibility of any statistical formula. Surely there will be guidelines to follow on how certain plays are coded, but it’s easy to be skeptical about the way they are going to determine how much credit/blame goes to each player.
 
I Need More Data!
The level of detail that goes into QBR has resulted in ESPN only having three seasons worth of data. They may want to hire some more people to work on previous seasons and get QBR done back to at least 2004.
 
Why 2004? That was the year the league reinforced the illegal contact rules, and you also had some key quarterbacks either emerge (Drew Brees) or enter the league as rookies (Eli Manning, Philip Rivers, Ben Roethlisberger, Matt Schaub). Only having three years of data makes it hard to see what kind of career arc one would expect from QBR. Is Josh Freeman’s 25.8 QBR in 2009 unusually terrible for a rookie, and how does Matt Ryan’s 72.6 in 2008 compare to Ben Roethlisberger’s rookie season? How high are Manning’s 2004 and Brady’s 2007 seasons? We don’t know, because there’s not enough data.
 
No Opponent Adjustments
While they are adjusting for the situation, QBR does not adjust for the opponent, which obviously plays a significant part in how well a quarterback performs. Shredding the 2010 Houston Texans would not be the same as doing it against the Green Bay Packers, yet under QBR, it would be.
 
Opponent adjustments are tricky since you would have to wait until a certain point in the season (possibly the end of the season) to start using them effectively. However, it can be done, and QBR should look into adding it.
 
At the very least they must give fans a defensive QBR on their website. Just as every defense has a defensive passer rating in a season, every defense will have a defensive QBR. Cold, Hard Football Facts has shown over the years that defensive passer rating is a huge indicator of success. Green Bay and Pittsburgh, for example, finished No. 1 nd No. 2, respectively, in 2010. With defensive QBR, we can at least get some context into how a defense fares against quarterbacks during the season.
 
If Tom Brady has a 79.0 QBR against the Bills, and the Bills are allowing an awful 83.5 QBR on the season, then Brady actually played a little below the average of what his peers have done against Buffalo. If they are not going to adjust for opponent, then give us the defense’s QBR each week.
 
Garbage Time Is Hindsight Only
The problem with assigning a lesser weight to a “garbage time” drive is that we only know something was truly garbage time after the game is over.
 
When Michael Vick completed a 65 yard touchdown pass to Brent Celek against the Giants with 7:28 left to cut the deficit to 31-17 last year, there was no reason to think this was nothing more than a stat-padding play. The Giants were still going to likely win the game. This play would get a lesser score in QBR because of the situation (also because of a missed tackle and big YAC from Celek).
 
Once the rest of the game unfolded, we found out that touchdown was absolutely critical in sparking the huge comeback by Philadelphia. They went on to recover the onside kick and score three more touchdowns to win the game 38-31. None of it happens without that initial scoring drive.
 
The touchdown to Celek was likely one of Vick’s lowest scores for a touchdown using QBR, but the reality is it was one of the most important touchdowns of the 2010 season.
 
Quantifying “Clutch”
In my own research of comebacks and game-winning drives, one of the biggest problems revolves around what to do with quarterbacks that get behind big, and stay behind big. The only drives used in my data are ones where you have the ball in the fourth quarter/overtime and the game is either tied or down one score.
 
The problem is when a quarterback is down multiple scores, gets the game to within one score, then maybe only has one opportunity in the last minute to tie or win the game. If it doesn’t happen, he gets a failed comeback. Meanwhile the quarterback that started the fourth quarter down two scores never does anything to get it closer and finally loses by two (or more) scores. That quarterback does not get credited with anything because he never had the ball down one score. He prevents adding a loss to his comeback record by not doing anything to help win the game.
 
There is no fair way around this either. You have to keep the one score rule, because that at least keeps every offense on the same level, where no matter if there’s ten minutes or one second on the clock, you are one play away from tying the game (two if you need a two-point conversion).
 
A quarterback could really game the system by winning a few close games during the season, but not even coming close to that situation in the team’s losses. No penalty for not succeeding in higher-weighted situations, so his clutch index will look better than a quarterback that plays well enough to have his team close in every game, but may end up losing several times (Aaron Rodgers is a perfect example).
 
ESPN is supposed to have it average out where being in more clutch situations does not help your rating more than doing well in them, but the results suggest otherwise given the placement of Vince Young (2009), Matt Ryan (2008, 2010) and Josh Freeman (2010). It may also explain why Tom Brady’s 2010 season (76.0) ranks fourth in the last three years, even though his conventional passer rating is the highest in that time.
 
The Trent Dilfer Factor
ESPN would probably be best served to keep Trent Dilfer’s name away from this stat. If anyone thinks he has something to do with it, they will likely dismiss it as a former quarterback that was consistently below-average at passing (and just about everything else) trying to create a system that explains why he still had a 58-55 record as a starter and won a Super Bowl (hint: defense).
 
More than that, Dilfer’s example of QBR during the Friday night special of a game Philip Rivers had against the Dallas Cowboys on 12/13/2009 was especially difficult to understand.
 
In the game, Rivers completed 21/32 passes for 272 yards, 1 TD, 1 INT, 89.6 passer rating, 1 sack for 4 yards, 38 yards gained on defensive pass interference
 
Based on pure stats it’s a solid performance on the road against a playoff team. FootballOutsiders ranked it as the sixth best QB performance of week 14. According to Dilfer and QBR, this game scored a 96.3 for Rivers. Why? Rivers was 5/10 on third down, which is very good considering some were long situations, but overall it’s nothing incredible.
 
Remember, the highest possible score is 100, and Rivers was 96.3. If you study the play-by-play, there should be enough negative plays to drop that rating down lower. After all, Michael Vick’s explosive performance against the Redskins on Monday Night Football last year was a 99.8 in QBR.
  1. 1-10-SD 16 (8:31) P.Rivers pass short left to M.Tolbert to SD 14 for -2 yards (D.Ware) [A.Spencer]. Pass complete in the flat.
  2. 3-3-DAL 12 (11:05) (Shotgun) P.Rivers pass short middle to V.Jackson to DAL 11 for 1 yard (D.Ware) [S.Bowen].
  3. 1-10-SD 1 (2:13) P.Rivers up the middle to SD 2 for 1 yard (I.Olshansky; B.James).
  4. 2-9-SD 2 (2:00) P.Rivers pass incomplete deep left to M.Floyd. Pass incomplete on fly pattern; Jenkins closest defender at the San Diego 40.
  5. 3-9-SD 2 (1:54) (Shotgun) P.Rivers pass deep right intended for V.Jackson INTERCEPTED by T.Newman at SD 27. T.Newman to SD 27 for no gain (V.Jackson).
  6. 1-10-DAL 46 (:30) (Shotgun) P.Rivers pass incomplete deep left to M.Floyd (M.Jenkins). Pass incomplete on fly pattern at the Dallas 25 sideline.
  7. 2-10-DAL 46 (:25) (Shotgun) P.Rivers sacked at DAL 50 for -4 yards (S.Bowen).
  8. 3-14-DAL 50 (:18) (Shotgun) P.Rivers pass incomplete short left to K.Wilson. Pass incomplete on crossing pattern
  9. 3-6-DAL 40 (10:49) (Shotgun) P.Rivers pass incomplete deep left to M.Floyd. Pass incomplete fly pattern; Newman closest.
  10. 1-10-DAL 44 (8:15) (Shotgun) P.Rivers pass incomplete short left to A.Gates. Pass incomplete on crossing pattern; Brooking closest defender.
  11. 2-10-DAL 44 (8:10) P.Rivers pass incomplete deep left to M.Floyd (A.Ball). Pass incomplete on deep seam route at the goal line.
  12. 3-10-DAL 44 (8:04) (Shotgun) P.Rivers pass short right to D.Sproles to DAL 41 for 3 yards (B.Carpenter). Pass complete after Rivers was forced out of the pocket.
That’s roughly 12/35 plays in the game where Rivers likely should have been downgraded. Several of them were failed third down conversions on the Dallas side of the field that prevented San Diego from scoring more points.
 
I watched this game live, but admittedly don’t have a great memory of it. It is possible some of these passes were drops, though the Dallas play-by-play would usually indicate that, which it does not.
 
The most troubling part was when Dilfer said that the interception thrown by Rivers (see play series 3-5) was not a big deal because it was a long pass and Dallas did not score after it.
 
Hopefully QBR does a better job of calculating things than that, because this was a terrible interception by Rivers. It was poorly thrown out of his own end zone, and Dallas took over at the San Diego 27 with 1:46 left in the half. Just because the San Diego defense held and Nick Folk missed the 42 yard field goal, that does not excuse the poor interception. The net expected points for Dallas had to be pretty good after starting that deep in Charger territory. What happened several plays later (the missed FG) should have zero statistical relation to the interception itself, which clearly put San Diego in a bad situation. Hopefully Dilfer simply misspoke.
 
Will It Catch On?
 
Only time will tell. If the casual fan does not like conventional passer rating because of the math involved, then they may be even more against QBR. Knowing ESPN, they will put as much marketing and self-promotion into QBR as they can. Expect to see QBR added to the stats page and box scores on their website, and to be mentioned in game recaps on their shows.
 
The Monday Night Football broadcasts will likely be a huge target for promoting QBR. Don’t be surprised if you turn on the game between Indianapolis and Tampa Bay on October 3, and hear a pre-game conversation awfully similar to this:
 
Mike Tirico: The Colts enter tonight’s game with a 1-2 record, and their leader, Peyton Manning, has seemingly been off a bit as he missed the entire training camp and preseason. Jaws, what has been going on with Manning’s game? And please explain why he’s actually played much better than the stats suggest.
 
Ron Jaworski: You’re right, Mike. Football fans always expect the stats to be there for Peyton Manning, and so far this season, they’re not up to the level you’re used to seeing from Peyton. However, those are just his conventional stats. They don’t measure a lot of important stuff that a quarterback has to deal with. Fortunately, ESPN has created this new Total Quarterback Rating, which uses the same game tapes I study all week, and when you measure a quarterback on everything he does in the game, Manning is still producing at a high level. He just needs a little more help from his teammates.
 
Mike Tirico: Right. Manning actually ranking second in the league in Total QBR through three weeks. Now Jon, the last time you and Peyton Manning were in this building on Monday Night Football, something unbelievable happened.
 
Jon Gruden: Well, I don’t know how to calculate his Total Quarterback Rating, but that guy, Peyton Manning, The Sheriff, was down by twenty one points against my Tampa Bay defense eight years ago with four minutes to go, and that’s when The Sheriff pulled out the big rifle and started shooting my guys apart all over the field. I think his QBR was 150 that night. I tell you, Peyton Manning is special. That guy is great. If I had The Sheriff, I’d still be coaching instead of sitting here with you knuckleheads talking about eight years ago. I love Peyton Manning.
 
Mike Tirico: Sorry Jon, I had to bring it up. We mentioned Manning number two in QBR. Well right behind him is the young Josh Freeman. Colts. Buccaneers. Should be a good one, coming up next.
 
If you get a case of déjà vu on October 3, remember where it came from.
 
A Better Solution
 
It is the Holy Grail of football statistics to be able to create a formula that produces one number that tells us everything about a player’s individual contributions to his team’s success.
 
The problem is it does not exist. Total Quarterback Rating is not the Holy Grail of quarterback stats. It is a number, with a lot of very interesting concepts behind it, but like any stat, it is still flawed. Frankly, the whole idea of finding one perfect stat is flawed.
 
Passer rating is a composite stat. Anyone can calculate the factors that go into it: completion percentage, yards per attempt, touchdown percentage and interception percentage. Can we calculate the components that make up QBR? No, and that is a problem.
 
While some people are content with one number, others want all the numbers. Show us the clutch index, the third down numbers, the drops, the YAC, the scrambling, the expected points, the sacks, etc. Show us the metrics for each facet of quarterback play that helped you arrive at this QBR number. It will help people better understand why that one number is what it is.
 
This also means creating new stats, or making lesser known stats more widely available. Why can’t a fan go on ESPN, NFL, SI, Yahoo, etc. and find stats for a quarterback throwing out of the pocket? We know they exist, but where?
 
Dropped passes? Give five people the game tapes for a team’s season, tell them to track the drops, and there’s a good chance you will get five different answers. They are that subjective.
 
What about sacks? Stats for how long the quarterback holds the ball would help. An average drop back time with splits (<3.0 seconds, >3.0, > 5.0) would go a long way in deciding how well an offensive line is playing for a quarterback. We know a quarterback like Roethlisberger avoids a lot of sacks, but why hasn’t anyone ever figured out how many he avoids in one season? 10? 30? 50? No idea.
 
With the video analysis, why not come up with stats for the type of routes the quarterback throws and his receivers run? Brett Favre was known to throw slants a lot. How many slants did he throw in 2009 with the Vikings? 25? 50? 100? If ESPN can track how many yards and at what speeds a guy can run on the field, then they should be able to provide route stats. It could also give some context to how often certain offenses have “schoolyard” offense where the quarterback scrambles, the receivers break their routes, and you have the potential for a big play that was not by design.
 
Total Quarterback Rating is a system that rewards quarterbacks for throwing accurate passes down the field rather than relying on screens, easy short passes, and YAC. It rewards them for strong performance in adverse situations such as third down, fourth down, the red zone, and close fourth quarter games. It does not penalize them for dropped passes or Hail Mary interceptions. It actually holds them accountable for fumbles, sacks and penalties.
 
It really sounds like a worthwhile system when you hear all of that. I have personally tracked many of those situations over the years for quarterbacks. The difference is I always kept them in separate files so their performance can be isolated for the situation. QBR mixes everything together in a completely unknown way to produce a single number. Some may like that, but others will not.
 
ESPN has to walk a fine line of still presenting fantasy football coverage, which relies heavily on volume and raw totals, and now they want to corner the market on situational stats. Time will tell how willing fans will be to accept what they have done. Not sure it will be the revolutionary change ESPN expects it to be, but it may be the start of drawing a lot of fans that previously had no interest over to the advanced statistical side of football.
 
There is no such thing as one perfect statistic. We can only try and perfect the stats we have. Getting more people to think about football as the strategic game it is would be the best outcome in all of this.

Visit ColdHardFootballFacts.com for more NFL articles

submit to reddit


By Scott Kacsmar
Contributor
Become a fan

Leave a Comment

Previous Comments (0)

Please to post a comment.

Connect With Us

Sign up for our newsletter to recieve all the latest news and updates...
Privacy guaranteed. We'll never share your info.