Ethan Nash 8/18/13 CS 221 Final Project

Autonomous Football Coach

Abstract: Play calling is an extremely valuable skill in the National Football League. Coaches get paid millions of dollars to be effective play callers and choosing the correct play can mean the difference between winning and losing in a game often decided by inches. For this project I created an autonomous football coach that calls plays based on the game situation. I used play-by-play data for every NFL regular season and playoff game between 2002 and 2012 (about 430,000 plays) to build a play-calling machine based on the effectiveness of different play calls in different situations. This data was made freely available by Brian Burke at Advanced NFL Stats. The autonomous football coach is modeled as a Markov Decision Problem where the states are the current game situation and the actions are different play calls. I chose this project because I love football and sports statistics. Sports statistics is often used to analyze past performances or to predict future performance. I think an autonomous football coach is an interesting way to use sports statistics because it lives in the present. I also thought it would be fun to compare my football coach with actual play calling tendencies in the NFL. Introduction: An effective autonomous football coach needs to take the game situation as an input and output a play call. Real coaches must deal with play books that have hundreds of different plays and the game situation can have a varying number of complex factors. In order to create an effective play calling machine, the first step was to simplify the game situation and play calls based on the play-by-play data I was using and some of my previous knowledge about football. Play calling is an interesting problem because there are so many factors a coach must consider when calling a play. I chose four factors for my autonomous coach: Down, distance to go for a first down, field position, and score differential. There are many other factors that a real coach must consider when calling plays: time remaining, strength of opposing defense, great individual defenders, game flow, momentum, and how well certain plays are working so far in the game, to name a few. Factors like game flow and momentum are difficult to quantify and factors involving the opposing defense was not available in the data I was using. I looked into using data from advanced football analysis websites like Football Outsiders and Pro Football Focus, but this data requires a membership fee to access their premium content. Time remaining was one factor that I could have used but opted to not include it because I was unsure how I would separate time remaining into different buckets and I did not want to dilute the data further from the factors I was already using. For play calls I chose to break plays into four groups: short pass, deep pass, run inside, and run outside. Plays can certainly be divided into different groups than this, however, this was the best I could do with my dataset. The play-by-play data gives a description of the play. For example, (10:20) E.Manning pass short middle to M.Bennett to DAL 47 for 9 yards (B.Carter). I preprocessed the data to

give categorize them into one of the four groups. If it did not fit, I ignored it. These include kickoffs, returns, spikes, kneels, field goals, and punts. Given the game situation factors and play call groups I chose, I was ready to make a few hypothesis based on my previous knowledge of football. Hypothesis: 1. In general a team is more likely to run the ball when it has fewer yards to go. 2. On third down the teams ultimate goal is to make a first down. Outside of very short yardage situations, this is hard to do by running the ball when the opponent might expect it. Because of this, the autonomous coach should typically pass the ball on 3rd and 3 and longer. 3. A team that is winning is more likely to run the ball in an attempt to protect the ball and run out the clock. Meanwhile, a team that is losing is more likely to pass the ball in an attempt to quickly score points to get back into the game. 4. Over the last several years the league is trending towards a more pass oriented league. In 2012, the average team amassed 3701 passing yards over the season, whereas the average team averaged 3396 in 2002. If using only the data for 2012 rather than for 2002-2012, it should be a more pass oriented autonomous coach. Method: In this section I will explain my model, algorithm, and interface of the autonomous football coach Model: I formalized the autonomous football coach as a Markov Decision Problem. States: A state is given by the game situation. This includes the down, yards to go, field position, and score differential. There are four downs (1, 2, 3, and 4), 15 yards to go buckets (1-14 and 15+), 10 field position buckets (goal-to-go, red zone, 20-29, 30-39, 40-49, 50-59, 60-69, 70-79, 80-89, and 90-99), and 3 score differential buckets (winning, tied, and losing). Actions: The autonomous coach has four possible play calls: short pass, deep pass, inside run, and outside run Transition Probability: The transition probability is a conditional probability table of the play call options given the game situation. For example, if a team has a first and 10 from the 80 yard line of a tie game (often the situation of the first play from scrimmage of a football game), the conditional probability of each play is as follows: Deep Pass Short Pass Run Inside Run Outside 12.71% 46.31% 19.70% 21.27% I chose a non-deterministic approach to try to model the unpredictability that coaches must show to be successful play callers. If a coach shows his hand then the defense can prepare for it. This is why coaches always cover their mouths with the playbook when they call plays. They dont want opposing mouth readers to be able to guess the play. For example, if in the above scenario the coach always called the play with the highest utility (short pass) then the defense would always defend against a short pass and the short pass would no longer have the highest utility. The Markov Model accounts for the utility of each play call while still incorporating randomness.

Algorithm: To learn the conditional probabilities of each situation I used a supervised learning algorithm. The data I got from advancednflstats.com had 18 columns for each row, where a row was a play. The columns were game ID, quarter, min, seconds, offensive team, defensive team, down, to go, yard line, score differential, series 1st down, play description, score change, next score, team win, offensive score, defensive score, and season. Most of this data was irrelevant for me. Using Excel VBA, I preprocessed this data into just down, to go, score differential, yards gained, first down gained, play type, turnover, and score. I then ignored the following play types: kick, punt, field goal, extra point, review, aborted, two point, kneel, penalty, and spike. For the rest of the plays, I assigned a utility based on how successful the play was. I gave each play .1 points per yard, -4 points for a turnover, 6 points for a score, and the current down points for a first down (i.e. 3 points if a first down was attained on a 3rd down). My system of assigning a utility for a play was a little arbitrary and is not perfect. I will discuss a better method of assigning utility in the conclusion as possible future work. After creating a utility for each play, I was ready to make the conditional probability table. For each game situation I added up the utilities of the four different plays and normalized them so that they added up to one. This gave me P(play call | game situation), which is exactly what I was trying to get. Interface: To display the data I created a graphical user interface using Tkinter. It allows the user to select a down, distance, field position, and score differential and then click the generate play button to select a play. In addition, it displays the probability table that it generates the random play from. Results: 1. Hypothesis 1 predicts that teams should run more in short yardage situations and pass more in longer yardage situations. To show this I averaged the probabilities of the down, field position, and score differential while holding yards to go constant. The results are to the right. The results shows that the autonomous football coach calls running plays steadily more frequently until about five yards, just as expected. Another expected trend shown is that as the yards to
0.7 Run Inside Run Outside Short Pass Deep Pass

Autonomous Coach GUI

Play Call Distribution over Yards to Go

0.6

0.5

Conditional Probability

0.4

0.3

0.2

0.1

5 Yards to Go

10

15

go gets higher, the coach calls more deep passes. 2. Hypothesis 2 predicts that the ultimate goal on third down is to get a first down. Therefore the team should only run in very short yardage situations and pass otherwise. The graph on the right shows a huge drop off in run probability after 2 yards. The short pass is the overwhelming favorite on third down. This is because deep passes have a lower completion rate so first downs are not as likely. However, deep passes do become more likely as the yards to go become greater.

Play Call Distribution over Yards to Go On Third Down 0.9 0.8 0.7 Run Inside Run Outside Short Pass Deep Pass

Conditional Probability

0.6 0.5 0.4 0.3 0.2 0.1 0

5 Yards to Go

10

15

Pass Percentage over Field Position 0.85

3. Hypothesis 3 predicts that teams that are losing will have a higher pass percentage than teams that are losing. The graph on the right shows the autonomous coaches average pass play percentage over different field positions. The graph shows that losing teams call pass plays more often across the board. On average, losing teams call a pass play on 73.6% of passes while winning teams call a pass play on 58.3% of passes.

0.8

Losing Winning

0.75

Pass Percentage

0.7

0.65

0.6

0.55

0.5

10

20

30

40 50 Field Position

60

70

80

90

4. Hypothesis 4 predicts that since the league is trending towards higher passing numbers than by using data from only 2012 we will have more pass play calls than using data from 2002-2012. The graph on the right shows the autonomous coaches pass play percentage over field position using the two different datasets. There seems to be no clear pattern between the two. However, I looked a little deeper at the autonomous coach and created another graph showing deep pass play percentage over field position, shown below. In this graph it is clear that the autonomous coach calls more deep passes when only using data from 2012. Deep

Pass Percentage over Field Position 0.7 2002-2012 2012

0.68

0.66
Pass Percentage

0.64

0.62

0.6

0.58

10

20

30

40 50 Field Position

60

70

80

90

passes have a higher average yardage per attempt so this explains why the league has seen higher passing numbers recently.
Deep Pass Percentage over Field Position 0.24 0.22 0.2

Deep Pass Percentage

0.18 0.16 0.14 0.12 0.1 0.08 0.06 0.04 2002-2012 2012

10

20

30

40 50 Field Position

60

70

80

90

Conclusion: The autonomous football coach provides a fairly accurate representation of basic NFL play calling strategies. However, there was a lot of things that I did not do because of a lack of data that I think could be done to improve the program. The biggest thing I wanted to do was incorporate Win Probability Added (WPA). WPA is a statistic that has been around for years in baseball but is fairly new to football. It is a statistic that looks at the current game situation (down, distance, field position, score, and time) and gives the probability that the offensive team will ultimately win the game. It is based on years of data from football games. In my model I give a utility score based on the outcome of a play. This is arbitrary, however, and certainly not perfect. A better model would compare the WPA of a game situation before and after the play to give a utility of a play call. While WPA calculators are freely available on the internet, I did not know how to incorporate it into my Python code, despite my best efforts. Another improvement that I would like to make as future work is to incorporate defensive strength into the model. Football Outsiders has a great stat called Defensive Value Over Average (DVOA) that compares a teams pass and rush defense against the average team. It would be cool if the autonomous coach was more likely to call a pass play against a team that had a poor pass defense, for example. Lastly, I think it would be great if the autonomous coach incorporated data from the current game and have that data be weighted to count for more than historical data. This way the coach could learn what is working well or not well on a particular day in order to call better plays.