Scouting Reports
Introduction
One responsibility NFL scouts have is to take notes on prospects. Here are some plausible snippets:
This quarterback sees ghosts.
Meaning the quarterback feels pressure when there is none.
This cornerback is a catch tackler.
This means the player waits for the ball carrier to initiate contact.
This player helped his backup scheme after he was injured.
Observing a player that’s been taken out of a game can still tell you information about his character.
We hypothesize that these scouting reports contain untapped information. Scouts may also assign grades to players based on their observations, this can be an overall grade or for specific attributes such as throwing accuracy. We aim to show that there is information in the reports that isn’t captured in the grades. We test this by checking if there is predictive power from the text for the success of the prospect.
Motivation
Many scouts have years of experience and can detect minor differences between players. Analytical methods have focused on predicting player success from statistics, like college stats and combine measurements. Significant advancements happen when different disciplines collaborate, in this case: sports, stats, and natural language processing.
We combine statistics and scouting reports for two ends:
- Improve predictions on players
- Uncover over and under valued features in prospects.
Our goal is NOT to replace scouts! That would waste information.
Data Collection
We collected data from 2010 to 2015. 2010 was the first year both grades and reports were available for players. 2015 was the last year allowing most players enough time to receive a second contract or be cut from the league.
We used data from the following sources:
- Scouting Reports: NFL Draft Tracker
- Historical Contracts: spotrac and Over The Cap
- Approximate Values: Pro Football Reference
Model
Input
The scouting report is the input which contains their grade and writeup about the players.
Output
Predicted average annual salary of their second contract. If a player does not sign a second contract then their salary label is zero. To get more samples, we use 5th year options as second contracts if they haven’t signed anything else. If a player is cut and hasn’t been signed again, we assign a salary of zero as well.
Scouting Data
From this we extract:
- scout grade
- NFL Combine stats
- height, weight, arm length, hand size
- n-grams from written overview, strengths and weaknesses
Salary Data
We retrieve salary data from spotrac.
Note: we correct for inflation by normalizing the salary.
Results
We fit a model using machine learning. We used feature importances which measures how useful the feature was in predicting the contract. We made a word cloud and visualize the terms given to the model based on their feature importance. Each term is colored as follows:
- Strength: if the term appeared in the “Strength” section.
- Overview: if the term appeared in the “Overview” section.
- Weakness: if the term appeared in the “Weakness” section.
- Unused: if the term had zero importance to the model.
Quarterback
Now let’s take a closer look at the term pocket strength.
The quarterbacks labeled in blue had the scout write something about them related to the pocket for a strength. Usually about how the quarterback is able to escape the pocket or how they have poise in the pocket. For example one scout wrote about Sam Bradford.
[Bradford] Has quick feet to sidestep the rush and the toughness to stand in the pocket and take a hit.
If we only look at Quarterbacks earning more than $7 million per year, about twice as many quarterbacks have a mention of pocket in their strengths then those that don't. This indicates that this scout is undervaluing the value of good pocket presence.
Wide Receivers
For wide receivers we’ll take a closer look at the size weakness.
Not surprisingly when scouts mention size as a weakness in wide receivers they say things like “only average size” or “size is a concern in the NFL”. For DeAndre Hopkins they wrote:
Only average size for a starting outside receiver…
From the chart there are 28 receivers with size mentioned as a weakness. However about half of those receivers earned more than $2 million per year on their second contract. This suggests that this scout is discounting undersized receivers too much. In fact, receivers had a better success rate with size as a weakness than receivers without.
Takeaways
- Everyone has biases including scouts!
- We are not replacing scouts!
- Scouting supplies data fundamental for this approach.
- We address debiasing scouting reports.
- We can give feedback to scouts to help them improve.
- We show that we can improve prediction of player success through market value.
