Professional Documents
Culture Documents
Seminar Report
Submitted
in partial fulfillment
Bachelor of Technology
Dept. of CSE
November, 2021
UNDERTAKING
I declare that the work presented in this report title “Blue Eyes Technology”,
submitted to the Computer Science and Engineering Department, Modern
Institute of Technology and Research Center, Alwar, for the award of Bachelor
of Technology degree in Computer Science and Engineering, is my original work.
I have not plagiarized or submitted the same work for the award of any other
degree. In case this undertaking is found incorrect, I accept my mistake.
i
ACKNOWLEDGEMENT
I would like to thank my institution and my faculty members without whom this
project would have been a distant reality, I also extend my heartiest thanks to my
family and well-wishers.
- Vaibhav Goyal
ii
TABLE OF CONTENT
UNDERTAKING...................................................................................................... i
ACKNOWLEDGEMENT ...................................................................................... ii
TABLE OF CONTENT
CHAPTER 1: INTRODUCTION .......................................................................... 1
1.1 Mobile Measuring Device or Data Acquisition Unit ................................. 2
1.2 Central System Unit ................................................................................... 3
CHAPTER 2: SOFTWARE DESCRIPTION ...................................................... 5
2.1 Connection Manager ................................................................................... 5
2.2 Data Analysis Module ................................................................................. 6
2.3 Visualization Module .................................................................................. 6
CHAPTER 3: EMOTION COMPUTING ............................................................ 8
3.1 Theory……………………………………………………………………..8
3.2 Emotion sensors…………………………………………………………...8
3.3 Emotion Sensors for hand ........................................................................... 9
3.3.1. Emotion Mouse ...................................................................................... 9
3.3.2 Sentic Mouse…………………………………………………….……10
3.4 Types of emotion sensors for Eye ............................................................. 11
3.4.1 Expression Glass .................................................................................. 11
3.4.2 Magic Pointing ……………………………………………………….12
3.5 Eye Tracking……………………………………………………………..12
3.5.1 Eye-attached tracking ................................................................................ 12
3.5.2 Optical tracking…………………………….…………………………….13
3.5.3 Electric Potential Measurement………………………………………….13
CHAPTER 4: ARTIFICIAL INTELLIGENT SPEECH RECOGNITION....15
4.1 Artificial Intelligence Technology ............................................................ 15
4.2 Speech recognition .................................................................................... 16
APPLICATIONS ................................................................................................... 18
CONCLUSION ...................................................................................................... 19
FUTURE SCOPE .................................................................................................. 20
REFERENCES ...................................................................................................... 21
CHAPTER 1: INTRODUCTION
Blue Eyes is a technology conducted by the Research Team of IBM at its
Almaden Research Institute (ARC), San Jose, California since 1997. Blue Eyes
Technology aims at creating a computer that have the abilities to understand the
perceptual powers by recognizing their facial expressions and react accordingly
to them.
Imagine yourself in a world where humans interact with computers. You are
sitting in front of your computer that can listen, talk, or even scream aloud. It has
ability to gather information about you and interact with you through special
techniques like facial recognition, speech recognition etc. It can even understand
your emotions at the touch of the mouse. It verifies your identity, feels your
presents, and starts interacting with you. You ask the computer to dial to your
friends at his office. It realizes the urgency of the situation through the mouse,
dials your friend at his office, and establishes a connection.
The blue eyes technology aims at creating computational that have perceptual and
sensory ability like those of human beings. It uses non-obstructive sensing
method, employing most modern video cameras and microphones to identity the
user’s action through the use of imparted sensory abilities. The machine can
understand what a user wants, where he is looking at, and even realize his physical
or emotional sates.
Blue eyes system monitors the status of the operator’s visual attention through
measurement of saccadic activity. The system checks parameters like heart beat
rate and blood oxygenation against abnormal and triggers user defined alarms.
1
Fig1.1: overall system overview
Blue Eyes system basically have two system i.e. mobile measuring device and a
central analytical system. Bluetooth module integrated with mobile device
providing wireless interface between sensors worn by the operator and the central
unit. Each of the operators is being assigned unique ID card and required user
profiles on the central unit side provide necessary data customization.
Blue Eye System basically consist of three device which are as follows:
2
Data Acquisition Unit is a Mobile part of the Blue eyes system is known as Data
Acquisition System. This system is used to fetch the physiological data from the
sensor and send it to the central system which is to be processed. Wireless
Bluetooth connections (connection establishment, authentication and
termination) must be maintained to fulfill this task. Personal ID cards and PIN
codes used to provide operator's authorization. A simple 5-key keyboard, a small
LCD display and a beeper is used to carry out communication with the operator.
The device uses exceptional situation to notify the operator. A small headset,
interfaced to the DAU with standard minijack plugs is used to transfer voice data.
3
The second associate of the wireless connection is Central System hardware. A
Bluetooth module (based on ROK101008) and a PCM codec is contained in this
Box which is used for voice data transmission. A parallel, serial and USB cable
are used to provide interfacing between this module and PC. The standard mini-
jack sockets provide the accessibility of audio data. A simple programming
device is being developed to program operator’s unique ID. Serial and PS/2
(power source) ports are being used to interface program to a PC. Inside, there is
Atmel 89C2051 microcontroller, which handles UART transmission and I2C
EEPROM (ID card) programming is being handled Atmel 89C2051
microcontroller which is present inside the Central System Unit.
4
CHAPTER 2: SOFTWARE DESCRIPTION
Looking after working operators' physiological condition is the main task of Blue
Eye System Software. Real time buffering of the incoming data, real-time
physiological data analysis and alarm triggering are being performed by the
software to show instance reaction on Operator’s condition. Several functional
modules System core is consisted in
The Blue Eyes software comprises several functional modules system core
facilitates the flow of transfer between other system modules (e.g. transfers raw
data from the Connection Manager to data analyzers, processed data from the data
analyzers to GUI controls, other data analyzers, data logger etc.).
5
• Communication with the CSU hardware
• Searching the new devices in the covered range
• Establishing Bluetooth connection
• Connection authentication
• Incoming data buffering
• Sending alerts
Data analysis module performs the analysis of the raw sensor data in order to
obtain information about the operator’s physiological condition. The separately
running data analysis module supervises each of the working operators.
The visualization module can be set in an offline mode, where all the data is
fetched from the database. Watching all the recorded physiological parameters,
alarms, video and audio data the supervisor is able to reconstruct the course of
the selected operator’s duty.
6
• A pie-chart used to represent a percentage of time the operator was actively
acquiring the visual information.
• A VU-meter showing the present value of a parameter time series
displaying a history of selected parameters value.
7
CHAPTER 3: EMOTION COMPUTING
It has been shown that people view their computer as having a personality by
Dryer (1999). it is important to develop computers which can work well with its
user.
3.1 Theory
Selected works from Ekman and others on measuring facial behavior describe
Ekman’s Facial Action Coding System (Ekman and Rosenberg, 1997).
Participants attached to devices to record certain measurements including pulse,
galvanic skin response (GSR), temperature, somatic movement and blood
pressure is one of the Ekman’s experiments. The participants were instructed to
mimic facial expressions which corresponded to the six basic emotions. Six basic
emotions as anger, fear, sadness, disgust, joy and surprise are described him.
From this work, Dryer (1993) determined how to distinguish various emotional
states using physiological measures. GSR, heart rate, skin temperature and
general somatic activity (GSA) are some measures which are being used. There
are basically two different kind of data analysis. A multidimensional scaling
(MDS) procedure is the first analysis which is used to determine the
dimensionality of the data.
1. Emotion Mouse
2. Sentic Mouse
8
B. Types of Emotion Sensors for Eyes:
1. Expression Glasses
2. Magic Pointing
3. Eye Tracking
Active, smart computer system is one goal of human computer interaction (HCI).
Gesture recognition, facial recognition, eye tracking, speech recognition, etc.
could possibly include in this type of project. Touching is another non-invasive
way to obtain information about a person. Computers are used to obtain, store
and manipulate data by the people. The computer must start gaining information
about the user in order to start creating smart computers. Gaining user information
through touch via a computer input device, the mouse is one of the proposed
methods. An emotional state may be determined from the physiological data
obtained from the user. The emotional state is related to the task the user is
currently doing on the computer. In order to gain a sense of the user's personality
over a period of time, a user model will be built. The scope of the project is to
create a better working environment where the user is more productive by having
the computer adapt to the user.
The Sentic Mouse is an experimental inspiration which come from the work of
Peter J. Lang, Ward Winton, Lois Putnam, Robert Kraus and Dr. Manfred Clynes.
This provide the base for designing a tool for the measurement of the human
being’s emotional valence response. Any emotional assessment of stimuli, from
positive (associated with pleasure, liking and attraction) to negative (associated
with displeasure, dislike and avoidance or revolution) can be generalized as
Emotional valence. Through this experiment quantitative values can be applied to
emotions so that a predictive model for emotional theory can be obtained.
Peter J. Lang and others showed subjects a series of pictures and asked them to
self-rate their emotional response. While the subject is being testing through this
experiment heart rate and skin conductance are being measured by the Ward
Winton, Lois Putnam, and Robert. conducted a series of Sentic experiments was
conducted by Dr. Manfred Clynes in which data are being gathered from every
vertical and horizontal component of finger pressure. The main aim of these
experiments is to quantify human emotions and map them into predictive model
of emotion theory. The Affective Computing research group gave approval to
these three models to apply interaction between human being and computers.
Using a computer, an experiment was conducted to provide the affective stimulus
to the human subject which combined all three emotion studies using a computer.
In Dr. Clynes experiments to collect Sentic data an ordinary computer mouse was
connected with a pressure sensor. Simultaneously as the subjects viewed Lang's
affective picture database, IAPS, we monitored the various other bio sensors were
also monitored and connected including GSR and EKG as forgotten by the work
done by Winton, Putnam, and Krauss.
The three: Sentic data, heart rate, and self-assessment are the three measured
results which were compared, against each other as well as against the
theoretically predicted results to assess the subject's emotional valence for each
slide. The results of the preliminary stages of analysis recommend the capturing
of valence information by the Sentic mouse.
10
Fig 3.2: Sentic Mouse
11
3.4.2 Magic Pointing
Eye tracking is the process of measuring either the point of gaze (where one is
looking) or the motion of an eye relative to the head. An eye tracker is a device
for measuring eye positions and eye movement. Eye trackers are used in research
on the visual system, in psychology, in psycholinguistics, marketing, as an input
device for human-computer interaction, and in product design. Eye trackers are
also being increasingly used for rehabilitative and assistive applications (related
for instance to control of wheel chairs, robotic arms and prostheses). There are a
number of methods for measuring eye movement. The most popular variant uses
video images from which the eye position is extracted. Other methods use search
coils or are based on the electrooculogram.
Eye-trackers measure rotations of the eye in one of several ways, but principally
they fall into three categories:
The first type uses an attachment to the eye, such as a special contact lens with an
embedded mirror or magnetic field sensor, and the movement of the attachment
is measured with the assumption that it does not slip significantly as the eye
rotates. Measurements with tight-fitting contact lenses have provided extremely
12
sensitive recordings of eye movement, and magnetic search coils are the method
of choice for researchers studying the dynamics and underlying physiology of eye
movement. This method allows the measurement of eye movement in horizontal,
vertical and torsion directions.
Each eye has an LED light source (gold-color metal) on the side of the display
lens, and a camera under the display lens. The second broad category uses some
noncontact, optical method for measuring eye motion. Light, typically infrared,
is reflected from the eye and sensed by a video camera or some other specially
designed optical sensor. The information is then analyzed to extract eye rotation
from changes in reflections. Video-based eye trackers typically use the corneal
reflection (the first Purkinje image) and the center of the pupil as features to track
over time. A more sensitive type of eye-tracker, the dual-Purkinje eye tracker,
uses reflections from the front of the cornea (first Purkinje image) and the back
of the lens (fourth Purkinje image) as features to track. A still more sensitive
method of tracking is to image features from inside the eye, such as the retinal
blood vessels, and follow these features as the eye rotates. Optical methods,
particularly those based on video recording, are widely used for gaze-tracking and
are favored for being non-invasive and inexpensive.
The third category uses electric potentials measured with electrodes placed around
the eyes. The eyes are the origin of a steady electric potential field which can also
be detected in total darkness and if the eyes are closed. It can be modelled to be
generated by a dipole with its positive pole at the cornea and its negative pole at
the retina. The electric signal that can be derived using two pairs of contact
electrodes placed on the skin around one eye is called Electrooculogram (EOG).
If the eyes move from the centre position towards the periphery, the retina
approaches one electrode while the cornea approaches the opposing one. This
13
change in the orientation of the dipole and consequently the electric potential field
results in a change in the measured EOG signal. Inversely, by analysing these
changes in eye movement can be tracked. Due to the discretisation given by the
common electrode setup, two separate movement components – a horizontal and
a vertical – can be identified. A third EOG component is the radial EOG channel
which is the average of the EOG channels referenced to some posterior scalp
electrode. This radial EOG channel is sensitive to the saccadic spike potentials
stemming from the extra-ocular muscles at the onset of saccades, and allows
reliable detection of even miniature saccades.
Due to potential drifts and variable relations between the EOG signal amplitudes
and the saccade sizes, it is challenging to use EOG for measuring slow eye
movement and detecting gaze direction. EOG is, however, a very robust technique
for measuring saccadic eye movement associated with gaze shifts and detecting
blinks. Contrary to video-based eye-trackers, EOG allows recording of eye
movements even with eyes closed, and can thus be used in sleep research. It is a
very light-weight approach that, in contrast to current video-based eye-trackers,
requires only very low computational power; works under different lighting
conditions; and can be implemented as an embedded, self-contained wearable
system. It is thus the method of choice for measuring eye movement in mobile
daily-life situations and REM phases during sleep.
14
UNIT 4: ARTIFICIAL INTELLIGENT SPEECH
RECOGNITION
• First one is that, it includes the studying of though out processes of human
beings.
• second one is that it includes the representation of those processes via
machines (like computers, robots, etc.,).
AI can be defined as the behavior of a machine which do the same work as done
by human intelligence. AI brought smartness to the computers and make computer
more useful and less expensive than natural intelligence.
15
4.2 Speech recognition
“Speaker-independent" systems do not use training and the systems which use
training are called "speaker-dependent" systems. Voice user interfaces such as
voice dialing (e.g. "Call home"), call routing (e.g. "I would like to make a collect
call"), demotic appliance control, search (e.g. find a podcast where particular
words were spoken), simple data entry (e.g., entering a credit card number),
preparation of structured documents (e.g. a radiology report), speech-to text
processing (e.g., word processors or emails), and aircraft (usually termed Direct
Voice Input) are some of the application of voice speech recognition.
16
A central processor unit (CPU) controls the input circuits that are fed by the
ADCS. A large RAM (random access memory) stores all the digital values in a
buffer area. This digital information, which is used to represent the spoken word,
is accessed by the CPU to process word further. The frequency range of normal
speech lies between 200 Hz to 7 kHz. A telephone call has bandwidth limitation
of 300 Hz to3.3 kHz so the identification of any call is very difficult. Filters and
ADCs process the spoken words. Each of these words represented by binary
values which becomes a template or standard, against which the words could be
compared. These templates are stored in the memory. After the completion of
storing process, the system can go into its active mode and then identify the
spoken words. Each word is converted into binary equivalent and is stored in
RAM.
The computer performs the searching operation and compares the binary input
pattern with the templates. It generally happens that even if the same speaker talks
the same, there always a slight variation comes in the amplitude or loudness of the
signal, pitch, frequency difference, time gap, etc. So, there could not be a perfect
match between the template and binary input word. Statistical techniques are used
in the pattern matching process which designed for the best fit. The result of the
subtraction of binary input word’s values from the corresponding values in the
templates is stored and If both the values are same, the result is zero and there is
perfect match. If not, the subtraction produces some difference or generate error.
The match’s quality increase with decrement in the error. When the best match
occurs, the word is identified and displayed on the screen or used in some other
manner. The search process takes a considerable amount of time, as the CPU has
to make many comparisons before recognition occurs. This necessitates use of
very high-speed processors.
A spoken word may last only a few hundred milliseconds, but it translated into
many thousands of digital words so a large RAM is required. The matching of
alignment of words and templates should be correctly done on time before
computing the similarity score. This process, known as dynamic time warping,
recognizes that different speakers speak the same words at different speeds as well
as elongate different parts of the same word.
17
APPLICATIONS
1. One of the main advantages of speech recognition system is that it allows the
users to do multiple works simultaneously. So that user can concentrate on
observation and manual operations, and still having control on machinery by
voice input commands.
2. Military operations have another major application of speech processing,
controlling of weapons by voice is an example. Reliable speech recognition
equipment provides pilots to give commands and information to the computers
by simply speaking into their microphones—they don’t need to use their hands
for this purpose.
3. Another good example is a radiologist scanning hundreds of X-rays,
ultrasonograms, CT scans and simultaneously dictating conclusions to a speech
recognition system connected to word processors. The radiologist can focus his
attention on the images rather than writing the text.
4. In airline and hotel reservations voice recognition could be used on computers.
A user only required to state his needs, to make reservation, cancel a
reservation, or doing enquiries about schedule.
5. Provide prevention from dangerous incidents
6. Brought decrement in ecological consequences financial loss a threat to a
human life.
7. Blue Eyes system provides technical means for monitoring and recording
human operator’s physiological condition. The key features of the systems are:
• Visual attention monitoring (eye motility analysis)
• Physiological condition monitoring (pulse rate, blood oxygenation)
• Operator's position detection (standing, lying)
• Wireless data acquisition using Bluetooth technology
• Real-time user-defined alarm triggering
• Physiological data, operator's voice and overall view of the control room
recording.
18
CONCLUSION
The BLUE EYES technology makes the computer so much smart and intelligent
that it behaves like a human being. It makes the life of human being simpler by
providing more luxurious and user-friendly services in computing devices. The
day when this technology will create its space into your house hold, making you
lazier is very near and it may even reach your hand-held mobile device. In short
Blue Eyes Technology can be called as a technological prediction.
19
FUTURE SCOPE
1. In the future, ordinary house hold devices-such as television, ovens may be
able to do their jobs when we look at them and speak to them.
2. Future applications of Blue Eyes Technology are limitless.
20
REFERENCES
1. Chandani Suryawanshi T. Raju, Blue Eyes Technology S. Madhumitha, IJSRD
- International Journal for Scientific Research & Development.
2. Raghavendra Priyam, Rashmi Kumari, Dr. Prof Videh Kishori Thakur,
“Artificial Intelligence Applications for Speech Recognition”.
3. V. Malarmathi, Dr. E. Chandra, “A survey on Speech Recognition”
International Journal of Computer Trends and Technology (IJCTT).
4. Mr. Gaurav N. Mehta, Mr. Jimish K. Desai, Mr. Devang G. Chavda, "Blue
EyesHuman-Operator Monitoring System".
21