At the Daily Norseman, CCNorseman has been working on developing a set of metrics for determining the chances of NFL success for prospective draft picks at the quarterback position:
This past off-season I have been scouring current and past scouting reports to try to develop a metric that we can use to evaluate quarterback prospects. I started by developing a metric to evaluate the traits of successful quarterbacks. I cataloged the traits found in pre-draft scouting reports of an elite list of 25 successful quarterbacks that have been drafted since 1998, and based the metric on those traits that were most common among that pool of players. In other words, I attempted to answer the question, “What common traits did the most successful quarterbacks in the NFL have coming out of college?” Then I went back and re-evaluated the “success metric” based on excellent feedback from the readers here at the Daily Norseman. I also developed a second metric to evaluate the traits of quarterback busts. It was the same process, except that I catalogued the common traits of the 17 quarterback busts since 1998 and based the bust metric on those traits that were most common among those players. That led me to the final Boom or Bust metric, which you can also find in that second link (and is listed below). The last step in this process is what you’ll find here: verifying the accuracy of the metric. I have gone back and run the metric on quarterbacks drafted in the 1st round of past drafts to see how successful it would have been at predicting the future successes of those players. The short of it is: it’s more accurate than a random guess. It’s not fool-proof mind you, but over the course of seven drafts from 2004 through 2010, it would have accurately predicted which 1st round quarterbacks would bust and which would be serviceable or better 73% of the time. Why did I only go back to 2004? Well, I really wanted to use at least two scouting reports for every quarterback when testing the metric to ensure better accuracy, but the farther back in time I went, the harder and harder it was to find reliable scouting reports online. I wasn’t able to track down more than one reliable scouting report for the quarterbacks drafted in 2003 and earlier, so there really is no other reason than that. I stopped at 2010, because a quarterback needs at least 4 years in the league to qualify as a bust or not, and those quarterbacks drafted in 2011 and later haven’t had a full 4 years yet.
[…]
It’s worth pointing out that in this particular data set (2004-2010), the Bust Metric by itself was almost as accurate overall as the combined metric in predicting the future of these quarterbacks and was 68% accurate by itself (although they each had slightly different results on a per quarterback basis). The success metric by itself was a little less accurate, correctly predicting the future only 61% of the time. In any case listed below are the 19 first round quarterbacks drafted between 2004 and 2010, with their metric scores from their pre-draft scouting reports and pre-draft prediction. I have taken some leeway in assigning the outcome score to this. My biggest concern in all of this is to ensure that if the metric predicts the quarterback to be in the bust category that they truly are a bust. After that, we can end up splitting hairs all day about what makes a quarterback “average” or “successful” or not. In other words, if the metric predicts that a quarterback will be merely league average, but he turns out to be a successful one then I’ll still call it a win for the metric, because it didn’t predict that quarterback to bust. I think teams are mostly concerned with not having their 1st round quarterback bust (like JaMarcus Russell or Ryan Leaf), than whether or not they get a Jason Campbell versus Aaron Rodgers type. I have given each quarterback an outcome label of “yes”, “maybe” or “no”. A “maybe” label essentially means that the player has performed reasonably well, but still has enough time left in their career to qualify for their prediction label. In those cases, the quarterback receives half-credit for their outcome.
