VISUAL COMMUNICATION WITH PARALYZED PEOPLE USING
FACE DETECTION ALGORITHM
HARI M
Department of BME, GRTIET
prasanthhari373@gmail.com
Abstract—Motor Neuron Disease (MND)
is a medical condition where the motor
neurons of the patient are paralyzed, it is
incurable. It also leads to weakness of muscles
with respect to hand, feet or voice. Because of
this, the patient cannot perform his voluntary
actions and it is very difficult for the patient to
express his needs as he is not able to
communicate with the world. The proposed
system detects the eye blink and
differentiates between an intentional long
blink and a normal eye blink. The proposed
system can be used to control and
Communicate with other people. The
objectives of the system are: Capturing the
frame from the video using the system’s
camera initializes the execution of the
proposed system. We used the Python Library
called OpenCV to capture the image of the
patient, and we use the python
language to do the Communication with
paralyzed people using Eye Ball blinks.
Keywords—Open-CV, Motor Neuron
Disease (MND), Image processing, Face
detection, Eye detection, Eye blink
I. INTRODUCTION
To do the communication with paralyzed people
we use eye Face Detection Algorithm. The Face
Detection Algorithm then processes captured
video frames to give out the rectangular
boxed face. This output from Face Detection
Algorithm then gets processed using AdaBoost
Classifier to detect the eye region in the face. Eye
detected will be sent to check if there is any
movement of the eyeball. If it’s there, then this
movement will be tracked to give out the
combination the patient is using to express the
dialogue. If not, then the blink pattern will be
processed to give out the voice as well as the text
MEENAKSHI N
Department of BME, GRTIET
madhumeenakshi052@gmail.com
input with the respective dialogue. Use the enter
key to start a new paragraph. The appropriate
spacing and indent are automatically applied.
There are many methods introduced for
motor neuron disease patients to communicate
with the outside world such as Brain wave
technique and Electro-oculography. Loss of
speech can be hard to adjust. It is difficult for
the patients to make the caretaker understand
what they need especially when they are in
hospitals. It becomes difficult for the patients
to express their feelings and even they
cannot take part in conversations.
The proposed system detects the voluntary
blinks of the patient and accordingly sends
the message about the requirement to the
caretaker.ss
The system uses an inbuilt camera to capture
the video of the patient and with the help of a
facial landmark algorithm, it identifies the face
and eyes of the patient. The system then slides
a bunch of images one after the other on the
screen and the patient can choose to blink over
the image he wants, just to convey message of
his desires. The system identifies the blink
with help of eye aspect ratio and then sends a
message to the care taker of what the patient
wants and also the system initiates a call to
the care taker where in a voice is audible
saying what the patient wants.
Blink To Speak offers a form of independence
to paralyzed people. The software platform
converts eye blinks to Speak. Every feature of
the software can be controlled by eye
movement. Thus, the software can be
independently operated by paralyzed people.
Using the sofware, patients can record
messages,recite those messages aloud, and
send the messages to others.
A. Problem Statement:
Patients who lose the ability to speak and write
they can only contact the outside world
through human-computer interaction; e.g.
controlling brain waves or tracking eye
movements. Currently, brain wave-controlling
devices need to be worn by users, so they are not
convenient for people to use. There exists eye-
motion-based software which enables the MND
patients to write in the computer by using their
eye functions only. When they are away from PC
and lie on the bed, they cannot communicate with
care providers. With the goal of helping MND
patients on the bed to call for other people with a
simple and easy approach this research aims to
develop a real time video processing system,
which can successfully research aims to develop
a detect the eye blinks regardless the head
directions, day or night.
B. Proposed System:
The proposed project aims to bring out a solution
for the paralyzed people without any harm to
their body externally or internally. It overweighs
the previously developed prototypes in this field
because none of the components are in direct
contact with the patients body hence it definitely
will prove to be safer.
Cost effective: The main objective of developing
algorithm of a real time video Oculography
system is that to provide cost effective for those
people who cannot afford. The existing technique
for such patients to communicate is too costly.
Thus, it is necessary to design a system which is
affordable to common people which includes cost
effective components for designing.
Fast: There are few algorithms which are
developed for video Oculography system for
communication. The main objective of this
project is to develop an algorithm which is
extremely fast compared to the existing ones.
Accuracy: The main objective of this project is to
develop an algorithm which is more accurate
compared to the existing ones that can afford the
technology. Blink To Speak focuses on a
different demographic that is often ignored. Blink
To Speak is free and open source. The software
runs on a wide variety of low-end computers. The
only required peripheral is a basic webcam. Not
only is this software accessible to paralyzed
people, but paralyzed people of almost all
financial classes as well.
C. Scope of the Project:
1. To make the Paralyzed people to
communicate with us.
2. To make the Perfect Assistance for
Paralyzed People
3. To give all the services for the paralyzed
people after they communicate their needs via
blinks.
II. LITERATURE SURVEY
Now-a-days, the rapid growth of technology
has made our PC become outdated. The tasks
that once we used to do with PC are now being
handled by mobiles or other smart devices.
Introduction of network enabled devices
or IOT devices have led to advanced
home automation systems. However, the
usage is limited for people with physical
disorders as remote control of an
appliance becomes difficult. In this paper the
project is about for those people who are
suffering from Paralysis (As example,
Tetraplegia Patients) and the difficulties which
they face while controlling home appliances.
Tetraplegia Paralysis is brought about by harm
to the cerebrum or the spinal line this patient;
client needs to control the appliance. We try to
take care of their issue utilizing eye blink
sensor. An eye blink sensor is a
transducer which detects an eye blink, and
gives a yield voltage at whatever point the eye
is shut. This project is about eye blinking
for instance, in systems that monitor a
paralyzed human so that he/she can operate
home appliances, such as light, fan, Air
Condition and so on. Also, this is
connected with Android Smartphone
Bluetooth radio, so that patients can
communicate with others in case of
emergency via sending text SMS, just by making zero and ones combination of white and
blinking their eyes. black pixels. We are considering 68 point pixel
for face and 6 point pixel for eye lids. The Face
D. Taneral, 2016. Presents Drowsy driver Detection is done using AdaBoost Algorithm
detection system using Eye Blink patterns [8] that processes on the captured video frames which is
is based on monitoring the changes in the eye converted as gray scale image to give out the
blink duration. In addition to tracking the face rectangular boxed face. The output from Face
and the eyes to compute drowsiness index Detection Algorithm then gets processed to detect
Garcia has also presented a nonintrusive face. Eye detected will be sent to check whether
approach for drowsiness detection based on the eye is closed or open. The outputs are in the
computer vision. It is installed in the car and it is form of image display, text alerting, audio played
able to work under real operation conditions. In and message sent to the concerned person.
this an IR camera is placed in the front of the
driver, in the dashboard. In order to detect his
face and obtain drowsiness clues from their eyes
closure.
H.S. Prashanth, 2012. Assistance for paralyzed
using Eye Blink Detection [9], The main aim of
this paper is to design a real time interactive
system that can assist the paralyzed to control
Abdul Rahaman Shaik Journal of Engineering
Research and Application www.ijera.com ISSN :
2248-9622 Vol. 9,Issue 8 (Series -II) Aug 2019,
pp 63-69 appliances such as lights, fans etc. or Fig 1: System Architecture
by playing pre-recorded audio messages,
through a predefined number of eye blinks. A. Visual Communication
Image processing techniques have been
Visual communication is the practice of using
implemented in order to detect the eye blinks.
visual elements to convey a message, inspire
DeepBose, BRAC University, Bangladesh, change, or evoke emotion. It’s one part
2017. Home automation with eye blink for communication design—crafting a message
paralyzed patients [10], The constant demand that educates, motivates, and engages, and
to improve daily living standards for paralysis one part graphic design—using design
patients or general people serves as a motivation principles to communicate that message so
to develop newer technology. The tasks once that it’s clear and eye-catching. Effective
performed by big traditional computers are now visual communication should be equally
solved with smaller smart devices. The study appealing and informative.
here talks about the development of a blink
sensor device which used for automated home
designs for disable people. B. Image Processing:
III. MATERIALS AND METHODS Image processing is the process of
transforming animage into a digital form and
The design part includes the system architecture.
performing certain operations to get some
It explains the workflow of the system proposed.
useful information from it. The image
The architecture mainly explainshow the data is
processing system usually treats all images as
being modified. How it is being used. and how
2D signals when applying certain
the results vary with it. Capturing the frame from
predetermined signal processing methods.
the video using the system’s camera initializes the
execution of the proposed system. The frames are Types of Image Processing
converted into grayscale image and we are
There are five main types of image processing:
• Visualization - Find objects that are not
visible in the image
•Recognition - Distinguish or detect objects in the
image
•Sharpening and restoration - Create an enhanced
image from the original image
•Pattern recognition - Measure the various
patterns around the objects in the image
•Retrieval - Browse and search images from a
large database of digital images that are similar to
the original image
C. Face Detection
Face detection has progressed from
rudimentary computer vision techniques to
advances in machine learning (ML) to
increasingly sophisticated artificial neural
networks (ANN) and related technologies;
the result has been continuous performance
improvements. It now plays an important role
as the first step in many key applications --
including face tracking, face analysis and facial
recognition. Face detection has a significant
effect on how sequential operations will
perform in the application.
In face analysis, face detection helps identify
which parts of an image or video should be
focused on to determine age, gender and
emotions using facial expressions. In a facial
recognition system -- which maps an
individual's facial features mathematically and
stores the data as a faceprint -- face detection data
is required for the algorithms that discern which
parts of an image or video are needed to generate
a faceprint. Once identified, the new faceprint can
be compared with stored faceprints to determine
if there is a match.
Fig 2. Face Detection from video frame
image
D. Eye Detection
Eye tracking refers to the process of measuring
where we look, also known as our point of
gaze. These measurements are carried out by
an eye tracker, that records the position of the
eyes and the movements they make.Near-
infrared light is directed toward the center of
the eyes (pupil), causing detectable
reflections in both the pupil and the cornea (the
outer-most optical element of the eye). These
reflections – the vector between the cornea and
the pupil – are tracked by an infrared camera.
This is the optical tracking of corneal
reflections, known as pupil center corneal
reflection(PCCR)
An infrared light source (and thus detection
method) is necessary as the accuracy of gaze
direction measurement is dependent on a clear
demarcation (and detection) of the pupil as
well as the detection of corneal reflection.
Normal light sources (with ordinary
cameras) aren’t able to provide as much
contrast, meaning that an appropriate
amount of accuracy is much harder to achieve
without infrared light.
THE CENTER OF THE EYE (PUPIL
CENTER) IS TRACKED IN RELATION
TO THE POSITION OF THE CORNEAL
REFLECTION. THE RELATIVE DISTANCE
BETWEEN THE TWO AREAS ALLOWS
THE CALCULATION OF THE DIRECTION
OF THE GAZE.
Light from the visible spectrum is likely to
generate uncontrolled Specular reflection,
while infrared light allows for a precise
differentiation between the pupil and the iris –
while the light directly enters the pupil, it just
“bounces off” theiris. Additionally, as infrared
light is not visible to humans it doesn’t cause
any distraction while the eyes are being
tracked.
Types of Eye Tracking Technologies:
1. Screen Based Eye Tracking
2.EyeTrackingglasse
Fig 3: Screen Based Eye Tracking
Fig 4. The Relative Difference In Location Of
The Pupil Center And Corneal Reflection Allows
For Deduction Of The Gaze Direction.
IV. BLINK DETCTION
Blink detection is actually the process of using
computer vision to firstly detect a face, with eyes,
and then using a video stream (or even a series
of rapidly-taken still photos) to determine
whether those eyes have blinked or not within a
certain timeframe.
Steps for Implementation
Step 1: Capturing a video
Step 2: Capture images from video
Step 3: Converting images into grayscale
Step 4: Fix landmarks on the images
Step 5: Detect Blinks
Step 6: Detecting Eye-ball movements
Step 7: Converting to text
Step 8: Sending the text (or) if emergency
situation means doing calls
Fig 5. Output image
V. CONCLUSION
Although blink detection systems exist
for other purposes, an implementation of a
blink detection system with the end use of
controlling appliances has not been previously
accomplished. While the system is intended
to assist the paralyzed and physically
challenged, it can definitely be used by all
types of individuals. The main challenge
involved in the implementation of the system
is the development of a real time robust blink
detection algorithm. Many algorithms have
been developed to serve the purpose, with
some being more accurate than the others.
This paper presented a blink detection
system based on online template matching.
The first phase involved the blink detection
phase; the second phase involved the counting
of blinks and subsequent control of appliances
through a micro controller. By enabling the
paralyzed to gain control of albeit a small part
of their lives, the system can offer some level
of independence to them. The helpers who are
assigned the task of tending to paralyzed
persons through the day can then be afforded a
break. The system needs moderate processing
power, making it suitable for practical use.
For continuous video input, laptops with
built in webcams or USB cameras will suffice.
VI. FUTURE ENHANCEMENT
The constant demand to improve daily living
standards for paralysis patients serves as a
motivation to develop newer technology. The
task once performed by big traditional
computers is now solved with smaller smart
devices. Paralysis is defined as the complete loss
of muscle function in any part of the body. It
occurs when there is a problem with the passage
of messages between the muscles and the brain.
The main objective is to design a realtime
interactive system that can assist the paralysis
patients to control appliances such as lights, fans
etc. In addition, It can also play pre-recorded
audio messages through predefined number of
eye blinks and it also helps to alert the doctor or
concerned person by sending SMS in case of
emergency by using eye blink Sensor. The eye
blink sensor is able to detect an intentional
blink from a normal blink, which is useful for
the paralysis patients especially Tetraplegic
patients to regulate their home devices easily
without any help. The system is limited by
the efficiency of the blink detection algorithm and
efficiency falls further under limited lighting
conditions. Since the initialization phase of
the algorithm is based on differencing between
consecutive frame background movement in
the frame may lead to inaccurate operation.
Typically, background movement causes non eye
pairs to be detected as eye pairs. This is overcome
to some extent by limiting the search region to
the face of an individual, by implementing a
face tracking algorithm prior to blink
detection. However, this in turn can lead to
reduced efficiency in blink detection. By giving
an option to the user to choose between the
system with and without face tracking, a level of
flexibility can be reached.
The application of the blink detection system is
not limited to the control of appliances but can
also be used for a variety of other functions.
Playback of audio distress messages over an
intercom system is one of the other
applications of the system.Future applications
of the system may include playback of video
or audio files by eye blinks and making a VOIP
call to play a distress message.
REFERENCES
1. Alberto J. Molina-Cantero, Clara
Lebrato-Vázquez, Manuel Merino-Monge,
Roylán Quesada-Tabares, Juan A. Castro-
García and Isabel M. Gómez-González
“Communication Technologies Based on
Voluntary Blinks: Assessment and Design”
Received April 10, 2019, accepted May 20,
2019, date of publication May 27, 2019,
date of current version June 11, 2019
2. Chu-Lian Xu and Chyi-YeuLin “Eye-
Motion Detection System for MND
Patients” 2017 4th IEEE International
Conference on Soft Computing and
Machine Intelligence
3. AsfandAteem, Mairaj Ali, Zeeshan Ali
Akbar, Muhammad Asad Bashir “Eye
Monitored Device for disable People”
2017 20th International Conference of
Computer and Information Technology
(ICCIT).
4. Milan Pandey, Anoop Shinde, Kushal
Chaudhari, DivyanshuTotla, Rajnish
Kumar, Prof. N.D. Mali “Assistance for
Paralyzed Patient Using Eye Motion
Detection” 2018 IEEE
5. Mohit Agarwal, Raghupathy Sivakumar
“Blink: A Fully Automated Unsupervised
Algorithm for Eye-Blink Detection in EEG
Signals” 2019 57th Annual Allerton
Conference on Communication, Control,
and Computing.