Individual Evaluation Proposal

Evaluation Proposal for Alabama Football’s Playbook Quiz Software for Game Plan

Retention

Matthew Hard

INTE-536
 I. Executive Summary

The University of Alabama Football Team relies on playbook retention in order to

traverse the college football season successfully. Football relies on players learning

their assignments and carrying these plays out every play for 70+ snaps a game.

Currently, there are over 40 formations, each with multiple variations on each play on

offense alone. Players are expected to study and learn these playbooks daily outside of

allotted football time.

To help serve the players, the team has subscribed to an XOS playbook quiz

system to allow the players to learn their playbook through electronic quizzes that can

be delivered directly to their tablets. This program seems beneficial to the players who

use it to retain their information in another format than just memorizing diagrams. Other

stakeholders such as administration are concerned with the effectiveness with the

program when it costs a substantial yearly subscription fee for the software. It is

important that this software and its implementation is evaluated in order to see an

appropriate use of school resources to implement this program.

The success of this software can be measured through quiz scores broken down

by position and quiz type. Surveys will also need to be conducted to compare the

learned plays from previous years to current years. This data can also be processed

through the use of Pro Football Focus data to accrue certain stats such as blown

coverages, offensive and defensive efficiency, and pre-snap penalties in order to

correlate learning pre and post XOS quiz software.

 II. Background and Context

The XOS Quiz software has been used by Alabama Football for 2 seasons, since

the summer of 2018. It has been used daily during the season each year we have had

it. According to the admin portal, 12,543 quizzes have been run since August 1, 2018 in

the system. The grades are collected by offensive and defensive GAs regularly, and

used to track how well players are learning concepts from the week. Before this

program, quizzes were made weekly by hand, and done on Friday nights in the hotel

the night before the game. With the introduction of this software, the coaching staff has

been able to track player learning throughout the whole week. This new program also

allows players to choose quiz topics that cater more towards what they are trying to

learn at the time, and not just a general quiz that is made for an entire position group of

players.

There are many variables that will go into deciding how effective this program is and

evaluating results. Overall, coaches want to see a reduction in errors on the field. This

can include but is not limited to pre-snap penalties, brown coverages, missed blocking

assignments, and general mental errors. The theory is that the better the players

perform on these quizzes, the better they will perform on the field.

Years like 2019 where more players graduated and left the team could skew the

numbers because there are more young guys who do not know the system as well who

could pull down the scores. There are also factors to consider like the changing of

coordinator positions where players would be required to learn new playing schemes.

All these factors will need to be taken into account in order to efficiently evaluate this

program.
 III. Descriptions and Definitions

Here are some terms used in this proposal that might not be familiar to those who

are not very familiar with the world of football, or any workings of intercollegiate

athletics.

Coordinator (Offensive/Defensive/Special Teams) – These are the individuals in

charge of their units. They determine what play style a team takes and how they will

work on the field.

Video Coordinator – These people are in charge of coaching video services for the

team as well as electronic playbook and quiz materials.

Assignments – In football assignments are what players are supposed to do on a

certain play. For an offensive player this might mean a certain route they need to run, or

a defensive player it might be a certain area of the field they are supposed to cover.

Assignments are what they players should be learning with this quiz system.

Pro Football Focus (PFF) – This is a company that aggregates data for all types of

football games. Alabama, along with most schools and all pro teams in the country,

relies on PFF data daily to analyze opponent film as well as to self-scout Alabama film.

PFF data will play a big role in evaluating player performance on the field in correlation

with their quiz scores.

IV. Stakeholders

a. Primary Stakeholders

i. Athletics Administration – Allocate funds to XOS Quiz system

ii. Head Coach – Wanting better daily performance out of players for

their assignments

iii. Catapult Systems – Catapult is the parent company of XOS and

wants to keep Alabama’s business.

b. Secondary Stakeholders

i. Video Coordinator – Responsible for operating XOS quiz software

daily. They are also the administrators of all iPad technologies

distributed to the players, and are therefore responsible for the end-

user experience of the quiz software.

ii. Position Coaches/GAs/Analysts – This group is responsible for

giving guidelines to the XOS quiz system in order to generate these

quizzes. They hold the key to creating effective assessment of the

player’s knowledge.

c. Tertiary Stakeholders

i. Players – Wanting to learn their assignments more efficiently and

perform better in game scenarios

ii. Alabama Fans – We hope to effect this group by creating better

play on the field to provide an overall more successful product than

there is now.
 V. Resources

Our two biggest resources for quantifiable data will be PFF data and quiz results

from the last two years. Pro Football Focus will provide us the data needed to create a

metric that measures the performance on the field relating to preparation. This will mean

that many data sets will be taken into account, and hopefully some sort of performance

score can be calculated through the use of these data points. PFF already creates a

team-by-team efficiency rating, and through the use of their online portal, reports can be

created to put all appropriate data into a single layout that can be used for this

evaluation.

Similarly, XOS quiz scores can be charted week by week to get a sense of how well

players are using this software to learn their assignments. Weeks will have to be

separated in order to see if different schemes give players a harder time than others.

For example, if one team is a triple option, players might have lower scores on this

certain week of quizzes because they are less familiar with this type of offense. Another

factor I mentioned earlier would be to factor in the players that actually participated in

the game, and to see how much prior experience they had compared to how the team

performed in the game.

The other resource that we can lean on for this evaluation is surveys from

players and coaches to see how effective the quizzes are in their opinion. It would be

interesting to see if quantifiable data suggests a change and if the players and coaches

agree or disagree if they help at all.

VI. Evaluation Questions

a. How effective is the XOS quiz software at helping players to learn their

playbooks for games?

b. How much of an impact is the XOS quiz software having on play on the

field for the football team?

c. How much value is the team getting from XOS quiz software in order to

keep paying for it?

d. How effective is the XOS quiz software at teaching the playbook when

compared to the handmade tests used before?