March Madness — Analyze video

to detect players, teams, and

who attempted the basket
Doing cool things with data!

Priya Dwivedi @ Deep Learning AnalyticsFollow

Mar 29

Introduction
Its March madness month! And what an exciting season it has
been. For the data scientist within you lets use this opportunity
to do some analysis on basketball clips. With the use of deep
learning and opencv we can extract interesting insights from
video clips. See example gif below of the game b/w UCF and
Duke where we can identify all the players + referees, label
players into teams based on their jersey colour. Later in the blog
I show how we can also identify which player is attempting to
shoot the basket . And all of this can be done real time.
Detecting players and teams

You can find the code on my Github repo

So let’s start.

Detecting Players
I have used a pretrained detection model like the Faster
RCNN to detect players. It is easy to download a Faster RCNN
trained on the COCO data set from the Tensorflow Object
Detection API and test it. The API takes as input each frame of
image and detects among 80 different classes. If you are new to
Tensorflow Object Detection and want to learn more, please
checkout this blog. The model does quite well in detecting
persons but there are many detections in this video due to the
large number of people in the crowds. See sample detection
below. I suppressed detections that were too big to more cleanly
segment out players. You can also play with the score threshold
in the API to filter out low confidence detections. Checkout the
code on Github for tips on how to suppress boxes with low
scores and multiple false detections.

Detection output from Pretrained Tensorflow model

Detecting Teams
Now comes the interesting part. How do we detect which
players are UCF vs Duke? We can use OpenCV to do that.

If you are new to OpenCV please see the tutorial below:

OpenCV Tutorial

OpenCV allows us to identify masks of specific colours and we

can use that to identify white and black players. The main steps
are:

 Convert the image from BGR to HSV colour space.

 In HSV space specify colour ranges for white and black. This
takes a bit of experimentation and you can visualize the
impact of different thresholds in the notebook.
 Use OpenCV to mask(colour) pixels that are in the threshold
range.
 OpenCV Bitwise_and to colour black any pixels not in the
mask

See output below for white colour. They are masked as “pink”
with everything else in the background in black

Detecting white colour pixels

To identify team for each individual player we extracted

bounding box from tensorflow object detection and count the
percent of pixels in that bounding box that are non black to
decide the team for that player.

Overall code works quite well. However this is a hard coded

logic for identifying black and white jersey players. It can be
made more general by using clustering to find similar players

Detecting pose and who is shooting

OpenPose is a real-time multi person pose detection library. It
detects persons in an image and outputs keypoints for the main
joints for every person — can total up to 25 keypoints per person.
The code is open sourced. You have to install as suggested in
README here. Once installed you can run images through it
and get keypoints for all the players in a scene as shown below.
Open pose output

So how do we identify players attempting to shoot a basket?

We can look for players with their wrist keypoints above their
head. Implying hands raised. This could indicate ready to shoot
as in the scene above or could be defensive. Further the
coordinates of the ball along with those of the wrist keypoint can
be used to identify which player with raised hands has ball close
to them.

Conclusion
Deep learning has made is possible to do really cool analysis by
chaining different ideas. There is a lot of open source code and
pretrained models that you can use on your data to get started.
Above is just the starting point. Other cool things that can be
done include:

1. Using OCR to read the game scores to allow your system to

understand which team is winning
2. Tracking the ball to predict when a shot can score points
3. Tracking players to get stats for each individual players
4. Detecting events like a slam dunk, 3 point basketball etc.

Hope you pull the code and try it yourself.

I have my own deep learning consultancy and love to work on

interesting problems. I have helped many startups deploy
innovative AI based solutions. Check us out at —
 http://deeplearninganalytics.org/.

You can also see my other writings

at: https://medium.com/@priya.dwivedi

If you have a project that we can collaborate on, then please

contact me through my website or at
info@deeplearninganalytics.org

References

 Tensorflow Object Detection API

 Good tutorial on Detecting colours using OpenCV