Machine Learning and Big Data at Foursquare

Blake Shaw, PhD Data Scientist @ Foursquare @metablake

At foursquare, we think there is a great opportunity to leverage massive amounts of location data to help people better understand and connect to places

What is foursquare?
An app that helps you explore your city and connect with friends A platform for location based services and data

So, what is foursquare? Its an app that helps you explore your city and connect with friends. Its also a platform for people to build location based services and collect and share location data

What is foursquare?
People use foursquare to: check in to places discover new places share w/ friends get tips about places get deals earn points and badges keep track of visits

People on foursquare check-in on their phones when they get to a place, to nd out more about it, share that they are there with friends etc.

What is foursquare?
Mobile Social

Local
Foursquare is in a unique place, sitting at the intersection between mobile, social, and geo.

Stats

10,000,000+ people 25,000,000+ places 1,000,000,000+ check-ins 10,000+ actions/second

Foursquare is generating a ton of data, every second 35 people check-in to a location. This data offers an unprecedented view into the behavior of millions of people worldwide, as they move around cities.

Growth

Here we see the growth of the service over the last two years since, it started in mid 2009.

Growth

Growth

Foursquare now has data on over 25 million places all over the world.

Learning with location data Check-ins are a rich source of data that
describe human behavior

We apply machine learning algorithms to

the collective movement patterns of millions of people to build exciting new services

Check-ins are a rich source of information describing human behavior. We apply machine learning algorithms to the collective movement patterns of millions of people to build exciting new services. We use a variety of ML algorithms, collaborative ltering, pagerank, clustering, classication and regression

Recommendation engine
foursquare explore
provides realtime recommendations using: location time of day check-in history friends preferences venue similarities

For example, last year we launched foursquare explore, a recommendation engine that uses a variety of signals to recommend places in real time that a user might be interested in. Explore uses a variety of machine learning models to rank venues. We combine many signals, including: the location of the user, and the time of day the persons past check-in history, the places their friends check in the similarities between different venues

Signals about places

Consider these signals about places. Each place has a different signature based on who is coming to the place, when, and for how long. This plot shows 3 different places: Gorilla Coffee, Grays Papaya, Amorino (a restaurant) See how Gorilla Coffee is busy more in the morning, where Amorino is busy in the evening. Grays Papaya clearly has a strong lunch crowd, but also a late night peak on the weekends. How can we use machine learning to learn from these signals which places are similar?

Networks of people

We also have unique signals that describe people. Which people are friends. Who is checking in together. Etc. From checkins we can build a large colocation network that can be used to better understand how people interact with each other in the real world. Here we see an example of graph embedding applied to the foursquare employee network. People are placed near each other in 2D if they often colocate at similar places.

Networks of people
Brooklyn

Manha-an

SF

Australia
Different parts of this map line up to the different places in the world where foursquare employees live. This plot was made by applying minimum volume embedding, a non-linear graph based dimensionality reduction algorithm, to the foursquare employee network. Each person on this map can be described by thousands of numbers, indicating how often they visit different places. The goal is to reduce the dimensionality of this space to 2D while preserving the strong pairwise relationships.

Open questions How to measure similarity between people

and places? How to determine inuence in large networks of people and places? What statistics can we use to describe peoples behavior in the real-world? How do we predict what information will be timely and relevant to a user?

We are constantly considering the best ways to address many of these questions...

Our data stack MongoDB Amazon S3, Elastic Mapreduce Hadoop Hive Flume R and Matlab

All of this is possible because of our world-class data stack. Amazon S3 and EC2 allow us on-demand access to huge computational resources

Join us!
foursquare is hiring! 85+ people and growing foursquare.com/jobs
Blake Shaw @metablake blake@foursquare.com

Thanks so much. Foursquare is hiring, if these projects seem interesting to you, please contact us at foursquare.com/jobs