An Automated App for Multimedia

recommendation based on Person's

mood using Neural Network

PROJECT GUIDE-
MR.KAPIL DEV SHARMA
ASSISTANT PROFESSOR


TEAM MEMBRS

Prajval Rasik - 1000012912

Ujjwal Sonkar - 1000012549
Mehul Tyagi - 1000012525
Divas Pratap Singh - 1000012468

 Index
Abstract---------------------------------------------- 4

Introduction ---------------------------------------- 5-6

Problem Statement----------------------------------- 7

Special features-------------------------------------- 8-9

Screenshots of code---------------------------------- 12-22

Model Architecture----------------------------------- 23-25

System Requirements------------------------------ 26

Conclusion------------------------------------------- 27

Future Scope------------------------------------------ 28

Refference ------------------------------------------- 29
 ABSTRACT
Nowadays, one of the major causes of anxiety and stress is an individual’s busy schedule. We
got no one to share our feelings and thoughts with. So, the best way to get relief from this is
to get someone who listens and understands us. In order to build a bot that is capable of
these things, a few steps should be taken into consideration. Some of those parameters
include detection of the mood of the user, asking about the day and listening and responding
accordingly, and suggesting to them some music/movies according to their mood. This
project basically satisfies the user’s emotional needs and helps the user to feel good about
life.

The project “Second Self” detects human emotion from an image which is one of the most
challenging research tasks in social communication. Machine learning-based emotion
detection gives performance better than traditional methods with image processing.
 INTRODUCTION
Second Self

The Second Self is an AI-based model which is designed not only for
entertainment/recommendation purposes but also acts as a mood
lifter.

The term "artificial intelligence" had been used to describe machines

that mimic and display "human" cognitive skills that are associated
with the human mind, such as "learning" and "problem-solving".

It will detect/predict the mood of a person and will then deal with a
person with different moods in different ways. The moods are
classified as sad, happy, excited, and angry. And then it would guide
and will give a positive approach to living life.
 PROBLEM STATEMENT

One of the significant causes of anxiety and stress is an individual’s busy schedule. We got no one
to share our feelings and thoughts with. So, the best way to get relief from this is to get someone
who listens and understands us.

In order to build a bot that is capable of these things, a few steps should be taken into
consideration. Some of those parameters include detection of the user's mood, asking about the
day and listening and responding accordingly, and suggesting to them some music/movies
according to their mood.

This project basically satisfies the user’s emotional needs and helps the user to feel good about life.
It detects human emotion from an image which is one of the most challenging research tasks in
social communication. Machine learning-based emotion detection gives performance better than
traditional methods with image processing.

 PROPOSED WORK
Capturing the mood –
People tend to express their emotions, mainly by their facial expressions. Capturing
and recognizing the emotion of the user and displaying appropriate songs, movies,
web series, blogs, etc. This project aims to capture the emotion expressed by a
01 person through facial expressions. Our application is designed to capture human
emotion through the camera and then with the help of image segmentation and
image processing techniques extracts features from the face of a target human being
and tries to detect the emotion that the person is trying to express.

Suitable recommendation –
Aggregating the data provided by the users, the system provides the user with a list
of recommended items they may be interested in based on the user’s mood.
02 Analyzing the data, our trained AI recommends movies, music, news, books, research
articles, search queries, social tags, products, jokes, and much more.
SCREENSHOTS
As you can see, we are given a very detailed analysis.
It gives us the following information:

The emotion recognition model is trained on FER 2013 dataset. It can detect
7 emotions

The bounding box coordinates the face in the image with the region
parameter.

The project works by getting live video feed from web cam, pass it through
the model to get a prediction of emotion.

Then according to the emotion predicted, the app will fetch playlist of songs
from Spotify through spotipy wrapper and recommend the songs by
displaying them on the screen.
 FLASK
What is Flask Python
Flask is a web framework, it’s a Python module that lets you develop web applications easily. It’s
has a small and easy-to-extend core: it’s a microframework that doesn’t include an ORM (Object
Relational Manager) or such features.

What is a Web Framework?

A Web Application Framework or simply a Web Framework represents a collection of libraries and
modules that enable web application developers to write applications without worrying about low-
level details such as protocol, thread management, and so on.
 FLASK
What Is Flask?
Flask is a web micro-framework developed by Armin Ronacher.
Flask is written in Python programming language.
Flask is made for Python applications.
Flask helps in implementing a machine learning application in Python that can be easily
plugged, extended and deployed as a web application.
Flask is based on two key components: WSGI toolkit and Jinja2 template engine. WSGI is a
specification for web applications and Jinja2 renders web pages.
Linkedin, Pinterest use the Flask framework.
 PROJECT COMPONENTS
Spotipy is a module for establishing connections to and getting tracks from Spotify using
Spotipy wrapper.
haar cascade is for face detection.
camera.py is the module for video streaming, frame capturing, prediction, and
recommendation which are passed to main.py.
main.py is the main flask application file.
index.html in the 'templates' directory is the web page for the application. Basic HTML and CSS.
utils.py is a utility module for video streaming of web cameras with threads to enable real-time
detection.
train.py is the script for image processing and training the model.
MODEL

Fig 1. mood detection model

Fig 2. mood detection model with users mood results

Joy Happy Anger Neutral Sadness

Haar cascade algo
Haar cascade algorithm for face detection

Used to identify images, objects, and videos.

Divided into 4 parts :

1. Haar features selection

2. Creating integral images
3. Adaboost training
4. Cascading classifiers
SYSTEM REQUIREMENTS
Hardware Requirements
• Intel Core i5
• Graphics Card 2 GB
• RAM 4 GB

Software requirements
• Windows Operating System
• Anaconda with Python3.6
• Keras
• Tensorflow
• Spotipy
• Tkinter (For testing)
• Flask
 CONCLUSION
Emotion recognition is a field that keeps advancing at tremendous speed. One of the most
important aspects of analyzing emotions is analyzing human faces. While the technology is still not
perfect, advanced emotion recognition models outperform humans when it comes to emotion
recognition in images.

While it is possible to build a custom model, for over 90% of users that is not necessary. The
libraries that do exist are open-source, can be used for both commercial and non-commercial
purposes, and allow users to perform emotion recognition using just a few lines of code.

Probably the most popular libraries for performing emotion recognition are DeepFace and FER. In
this project, we demonstrated how to use FER, and we also pointed out the advantages and
disadvantages of it.

In the next phase of the project, we will demonstrate how to perform emotion recognition on
videos. We will try to predict whether a student is interested in a particular lecture or not. This could
in the future become a very powerful tool that helps teachers, professors, and those in the
education industry better cater to the needs of their students and make education more effective.
 FUTURE SCOPE
Movies and Shows Recommendation -
In the next phase of our project, we will be recommending not just
songs from Spotify but different movies and tv shows using TMDB
01 APIs. The use-case would be same just different APIs need to be
integrated well.

More relevant and latest Recommendations –

Adding a feature that will update specified suggestions for better
and more recent recommendations, a specific day over a fixed
02 duration say every sunday and append it to database.
Thank you!