Chapter 1

INTRODUCTION

1.1 INTRODUCTION TO INTERNET ON THINGS

The Internet of Things (IoT) is the interconnection of uniquely identifiable embedded
computing devices within the existing Internet infrastructure. Typically, IoT is expected to
offer advanced connectivity of devices, systems, and services that goes beyond machine-
to-machine communications (M2M) and covers a variety of protocols, domains, and
applications. The interconnection of these embedded devices (including smart objects), is
expected to user in automation in nearly all fields, while also enabling advanced
applications like a Smart Grid. Things, in the IoT, can refer to a wide variety of devices
such as heart monitoring implants, biochip transponders on farm animals, electric clams in
coastal waters, automobiles with built-in sensors, or field operation devices that assist fire-
fighters in search and rescue. Current market examples include thermostat systems and
washer/dryers that utilize wifi for remote monitoring.

According to Gartner, Inc. (a technology research and advisory corporation), there

will be nearly 26 billion devices on the Internet of Things by 2020. ABI Research estimates
that more than 30 billion devices will be wirelessly connected to the Internet of Things
(Internet of Everything) by 2020. As per a recent survey and study done by Pew Research
Internet Project, a large majority of the technology experts and engaged Internet users who
responded—83 percent—agreed with the notion that the Internet/Cloud of Things,
embedded and wearable computing (and the corresponding dynamic systems ) will have
widespread and beneficial effects by 2025. It is, as such, clear that the IoT will consist of a
very large number of devices being connected to the Internet.

Integration with the Internet implies that devices will utilize an IP address as a
unique identifier. However, due to the limited address space of IPv4 (which allows for 4.3
billion unique addresses), objects in the IoT will have to use IPv6 to accommodate the
extremely large address space required. Objects in the IoT will not only be devices with
sensory capabilities, but also provide actuation capabilities (e.g., bulbs or locks controlled
over the Internet).
 To a large extent, the future of the Internet of Things will not be possible without
the support of IPv6 and consequently the global adoption of IPv6 in the coming years will
be critical for the successful development of the IoT in the future. The embedded
computing nature of many IoT devices means that low-cost computing platforms are likely
to be used. In fact, to minimize the impact of such devices on the environment and energy
consumption, low-power radios are likely to be used for connection to the Internet. Such
low-power radios do not use WiFi, or well established Cellular Network technologies, and
remain an actively developing research area.

However, the IoT will not be composed only of embedded devices, since higher
order computing devices will be needed to perform heavier duty tasks (routing, switching,
data processing, etc.). Companies such as FreeWave Technologies have developed and
manufactured low power wireless data radios (both embedded and standalone) for over 20
years to enable Machine-to-Machine applications for the industrial internet of things.

Besides the plethora of new application areas for Internet connected automation to expand
into, IoT is also expected to generate large amounts of data from diverse locations that is
aggregated and very high-velocity, thereby increasing the need to better index, store and
process such data.

Diverse applications call for different deployment scenarios and requirement, which
have usually been handled in a proprietary implementation. However, since the IoT is
connected to the Internet, most of the devices comprising IoT services will need to operate
utilizing standardized technologies. Prominent standardization bodies, such as
the IETF, IPSO Alliance and ETSI, are working on developing protocols, systems,
architectures and frameworks to enable the IoT.

1.1.1 FUNCTIONING OF IOT

The Internet of Things is the expansion of the current Internet services so as to

accommodate each and every object which exists in this world or likely to exist in the
coming future. This article discusses the perspectives, challenges and opportunities behind
a future Internet that fully supports the “things”, as well as how the things can help in the
design of a more synergistic future Internet. Things having identities and virtual
personalities operating in smart spaces using intelligent interfaces to connect and
communicate within social, environmental, and user contexts. There are several fuzziness
about the concept of Internet of Things such as IoT can be broken in two parts Internet and
Things.

The worldwide network of interconnected computer networks based on a standard

communication protocol, the Internet suite (TCP/IP) while a things is an object not
precisely identifiable. The world around us is full of objects, smart objects and the existing
service provider known as Internet. The convergence of the sensors like smart objects,
RFID based sensor networks and Internet gives rise to the Internet of Things. With
increased usage of sensors the raw data as well as distributed data is increasing. Smart
devices are now connected to Internet using their communication protocol and continuously
collecting and processing the data.

Ubiquitous computing which was thought as a difficult task has now become a
reality due to advances in the field of Automatic Identification, wireless communications,
distributed computation process and fast speed of Internet. From just a data perspective the
amount of data generated, stored and processed will be enormous. We focused on making
this architecture as a sensor based architecture where each sensor node will be as important
as the sensor network itself. Visualizing each sensor as having intelligence is the ultimate
aim of any architecture in the IoT domain.

There is a lot of pervasive presence in the human environment of things or objects,

described general overview of internet evolution with several IoT services with the use of
radio-frequency identification (RFID) tags, sensors, actuators, mobile phones, smart
embedded devices, etc. which, through unique addressing schemes, are able to effectively
communicate and interact with each other and work together to reach a common goal of
making the system easier to operate and utilize. The objects that will be connected will be
adaptive, intelligent, and responsive
 Figure 1.1: functioning of internet of things (IoT)

1.1.2 FUTURE VISION OF IOT

The Internet of Things is a vision which is under development and there can be many stake
holders in this development depending upon their interests and usage. It is still in nascent
stages where everybody is trying to interpret IoT in with respect to their needs.

Sensor based data collection, data management, data mining and World Wide Web
is involved in the present vision. Of course sensor based hardware is also involved. A
simple and broad definition of the internet of things and the basic idea of this concept is the
pervasive presence around us of a variety of things or objects – such as Radio-Frequency
Identification (RFID) tags, sensors, actuators, mobile phones, etc. which, through unique
addressing schemes, are able to interact with each other and cooperate with their neighbours
to reach common goals.

Figure 1.2 has been discussion on three particular visions given by:
• Things Oriented Vision

• Internet Oriented Vision

• Semantic Oriented Vision

Things Oriented Vision

This vision is supported by the fact that we can track anything using sensors and pervasive
technologies using RFID. The basic philosophy is uniquely identifying any object using
specifications of Electronic Product Code (EPC) .This technique is extended using sensors.

It is important to appreciate the fact that future vision will depend upon sensors
and its capabilities to fulfill the “things” oriented vision. We will be able to generate the
data collectively with the help of sensors, and sensor type embedded system. The
summarized vision will be dependent upon sensor based networks as well as RFID-based
Sensor Networks which will take care of the integration of technology based on RFID and
sophisticated sensing and computing devices and the global connectivity.

Internet Oriented Vision

The internet-oriented vision has pressed upon the need to make smart objects which are
connected. The objects need to have characteristics of IP protocols as this is one of the
major protocols being followed in the world of Internet. The sensor based object can be
converted in to an understandable format, which can be identified uniquely and its attributes
can be continuously monitored. This makes the base for smart embedded objects which can
be assumed to be a microcomputers having computing resources.

Semantic Oriented Vision

This vision is powered by the fact that the amount of sensors which will be available at our
disposal will be huge and the data that they will collect will be massive in nature. Thus we
will have vast amount of information, possibly redundant, which needs to be processed
meaningfully. The raw data needs to be managed, processed and churned out in an
understandable manner for better representations and understanding.

If we are able to make the sets of data into homogeneous and heterogeneous formats
then the interoperability issues of understanding the data will be dependent upon the
semantic technologies to process the data. It is here that needs a generic vision of processing
the raw data in to meaningful data and a marked separation of data and their interpretation.
 Figure 1.2: Three main visions of Internet of Things.

Meaningful inferences to develop a unified picture of the situation for smart

applications. This is possible through large scale sensor networks, data processing and
inferences using smart sensing and cloud computing. The inferences will make the sensors
sensible enough to rectify the whole process which in turn will move towards making the
whole process intelligent. The components that make up the WSN monitoring network
include:

• Wireless Sensor Networks hardware

Typically a WSN node contains interfaces to sensors, computing and processing units,
transceiver units and power supply. More sophisticated sensor nodes can communicate over
multiple frequencies.

• Wireless Sensor Networks Communication Stack (WSNCS)

The nodes will be deployed in an adhoc manner. Communication topology will be an

important factor for communication through the system of WSN nodes. There is this
communication stack at one central node which will be able to interact with the connected
world through the Internet and which will act as a gateway to the WSN subnet and the
Internet.

• Middleware:-

This is associated with the internet infrastructure and the concept of service oriented
architecture (SOA) for access to heterogeneous sensor resources as described in. WSNs
technological advances in hardware domain catering to circuits and wireless
communications have made robust and cost effective devices in sensing applications.

This has led to the use of sensors in wireless communication devices in diversified
environments. Sensor data is collected and sent for centralized, distributed or any hybrid
processing module for data processing. Hence, there are several challenges WSN has to
face to develop a successful IoT communication networks.

Communication Mechanism:- 6lowpan Challenges

All the objects that are present in the environment can be called object fit to be the “things”
of the internet. All these objects need an address which must be unique. This uniqueness
property will be a unique constraint and it will pave the way to gather information and even
control sensor based devices. Internet Protocol is the standard based protocol which is used
for internetworking methods of Internet. The first version was IPv4 and was thought of
having huge address spaces. But IPv4 got exhausted. smart embedded devices or simply a
sensor. Their communication mechanisms will be Wi-Fi, DSL, Satellite, Cable, Ethernet
and so forth.

The typical packet size of the communicating protocol will be around 1500 data
bytes to 9000 data bytes and even more. Today large amount of spatial data is also being
generated and thus we can use to use metadata for connecting database and Internet. As
happens in World Wide Web, the operations with sensor nodes may not be possible by
giving unique names to the sensors. Instead a unique address scheme must be formulated
and will be known as Unique Resource Name (URN).A look up table of these URN must
be present at the centralized node commonly known as gateway to the sensor sub system.

Thus entire network now forms a web of connectivity from users (high-level) to
sensors (low-level) that is addressable (through URN) accessible (through URL) and
controllable (through Uniform Resource Characteristics - URC) [11]

Data Storage

As IoT is getting developed the amount of data getting created is huge. The data centers
which will be storing this data will also needs space requirement as well as the energy and
power resources. It is this data which needs to be organized and processed.
 Semantic data fusion models will also be required to create meaning out of this data.
Artificial Intelligence algorithms must be applied to extract meaning from this redundant
data. Data storage and analysis will be a challenge when the whole world will be connected
through IoT.

Visualization

Any interaction of user with the environment will need proper visualization software which
will highlight the sensing mechanism as well the interpretation of data scenario too. Touch
screens and smart embedded tablets have created a conductive environment for the system.
The information which is being processed in to meaningful data using sensor fusion
algorithms will present lot many inferences about the current situation.

1.1.3 IOT: SERVICES AND APPLICATIONS

Let us look into the possible set of future possibilities which we can have a rewarding
applications. Some of the attributes which can be considered while developing application
is highlighted in which says the network availability, bandwidth, area of coverage,
redundancy, user involvement and impact analysis. Figure 1.3 mainly focuses on the
properties of the RFID, sensors and 6lowpan communication networks based IoT services.

Figure 1.3 Applications and services of IoT

Tracking:- People, Inventory and Logistic

The basis of this tracking is indeed RFID tags which are placed on object, human beings,
animals, logistics etc. RFID tag reader may be used in all the intermediate stages for
tracking anything which has the RFID tag in it. This object position identification can be
smartly used to trigger an alarm, event or a specific inference regarding a specific subject.
Smart Environment and Enterprise Collection

In any work environment an enterprise based application can come up with the fact that it
is based on smarter environment. Here the individual or the enterprise may give data to
outside world on its own discretion. Smart embedded sensor technology can be used in
order to monitor and transmit critical parameters of the environment.

Common attributes of the environment are temperature, humidity, pressure etc.

Smart monitoring of soil parameters can allow informed decision making about agriculture
and increase production of food grains and prevent loss of crops. Water conservation is
huge topic of concern where droughts are frequent. To limit water wastage, smart
technology can be used in water conservation.

Smart Unit

Another IoT application which is making waves is the Smart grid and smart metering
technology. The energy consumption can be efficiently monitored in a smart home or in a
small office or even a locality. This model can be extended over a city for better load
balancing. The world is fast changing and now camera based surveillance is high in
demand. This surveillance will not only require image processing but also computer vision.
IoT which will be based on video processing is a new technological challenge to integrate
large computation with small embedded device. Smart homes can be developed where
things of daily use will be tracked using sensor enabled technologies.

Local, Global and Social Sensing

Imagine a scenario where each of the family members of the family have a RFID enabled
gadget and thus object tracking can result actually in human tracking. This can readily
happen in IoT where common mobile phones can be used for tracking human beings. There
can be various types of sensors based devices which can be used for such type of tracking.
This is whole process is known as local, global and social sensing. The object can be tracked
locally, globally and in any place, any time and over any network.

Healthcare Monitoring Applications

Imagine a scenario in a village where old age persons, infants, pregnant ladies etc. have
RFID enabled chips over their bodies to track their vital health parameters. Any unusual
activity on their part will raise an alarm or an alert in the nearby local medical assistance
home. For example, RFID chips can be implanted in patients in order to track their medical
history.

Sensor technology can be used in emergency response, and health monitoring

applications. The information can be used to give medical assistance to the needful person
and in case of higher abnormalities, the nearby efficient hospitals can be alerted and thus
the hospitalization costs can be reduced through early intervention and treatment. This is
the advantage of smart healthcare using IoT.

Traffic Monitoring

In any city in the world, traffic monitoring is an important part of the smart-city
infrastructure. Normal traffic to highway traffic all requires adequate information about the
support and logistics available on the highway and in turn the system can be made self-
reliable and intelligent. Any type of congestion on roads will ultimately lead to loss of fuel
and economic loss. Any foresight on traffic will always help to improve the whole system.

With number of WSN and Sensor enabled communications, an IoT of traffic will
be generated. This will be known as Traffic IoT (TIoT). The information collected from
TIoT can be presented to travelers. The traffic information will be dependent upon the
queuing model on roads and infrastructure of roads itself. This identification of critical road
points and present state of traffic information on all roads can be provided to the user.
However this traffic monitoring application needs to be secure to prevent any terrorist
attack frequent in urban cities.

1.2 Problem Statement

One of the most precious gifts to a human being is an ability to see, listen, speak and
respond according to the situations. But there are some unfortunate ones who are deprived
of this. Communication between deaf-dumb and normal person have been always a
challenging task. The proposed device is an innovative communication system framework
in a single compact device. We provide a technique for a blind person to read a text and it
can be achieved by capturing an image through a camera which converts a text to speech
(TTS). It provides a way for the deaf people to read a text by speech to text (STT)
conversion technology. Also, it provides a technique for dumb people using text to voice
conversion and the gestures made by them can be converted to text.
 Tesseract OCR (Online Character Recognition) is used to read the word for blind,
the dumb people can communicate their message through text and gestures which will be
read out by espeak, the deaf people can be able to understand others speech through text.
All these functions are implemented using Raspberry Pi.

1.3 Goals and Objective of the Project

The main goal of our project is to provide a standard lifestyle for deaf dumb and blind
peoples as normal ones. Through this device the visually impaired people can able to
understand the words easily. The vocally impaired people can communicate their message
through text and gestures. The deaf people can able to understand others speech from the
text that is displayed. This helps them to experience the Independent life.
Chapter 2

PREAMBLE

2.1 Existing System

In the earlier days, blind people can only read Braille script. Braille is a tactile writing
system used by people who are blind. It is traditionally written with embossed paper.
Now-a-days Braille user can read computer screens and other electronics support using
refreshable braille displays.

Figure 2.1: Braille System

In Braille system, the language will go from left to right across the page, just like printed
words. The symbols which represent each letter are prepared up of between one and six
dots based on the figure of six dots which we would pick up on a dice or a domino. Later
in the evolution there exists screen reader system which is a computer program that enables
the blind masses to interpret what is shown on the screen through speech.
The next technology which is beneficial to the BVI is Finger Reader. It is a
wearable device in finger. It helps the BVI to access the plain printed text. People who ware
this device, scan a text line with their finger and in a result they get an audio feedback of
the words and also a haptic sense of the layout. These senses may be the start or end of the
line, new line and so on. It also alerts the reader if he moves away from the baseline thus it
helps him maintain straight scanning.

Figure 2.2: Finger Reader

Deaf and dumb people used sign language to communicate with outside world in masses.
Sign language is a linguistic process which is employed for communication among the
normal people and differently abled people using gestures. Traditionally, gesture
recognition method was divided into two categories namely vision-based and sensor-based
method. In vision-based method, the computer camera is an input device for various
gestures of hands and figures. In sensor-based systems, gloves are used which can achieve
the accurate positions of hand gesture.

Figure 3.3 Sign Language

Lots of studies have been done on sensor-based approaches like gloves, helmets etc. But
wearing it continuously is not possible. Therefore, further work is concentrated on image-
based approaches.

2.1.1 Disadvantages of Existing System

Blinds want the assistance of other neighbors to use braille script which is time

consuming and finger reader is not a language independent system and limited for

English language.

It is not possible for all the masses to learn the sign language to understand

whatever is said through gestures. Therefore, the communication gap still exists

between deaf and dumb people.

Dumb people can simply tilt the message by sign language which could not be

understandable by other people.

System consisting solution for all the three disabilities does not exist.
2.2 Proposed System

In resolving the difficulties with visually and vocally impaired people we have used the
tiny credit card sized computer named raspberry pi. By this device we provide the solution
for blind, deaf and dumb people.
The proposed system consists of input – microphone to record voice modulation,
camera to capture image, keyboard to type a message and output – speaker and device
screen to display the texts and the images. The user can give reply as text message and the
device does text to speech (TTS) conversion. The output is obtained from a small and
powerful speaker. The image is captured through the camera and the reading of a text can
be achieved by text to speech (TTS) conversion. The device also recognises the gestures of
the users and will displays the words related to it. Then the device does speech to text (STT)
conversion and displays it on the device screen, based on what the normal person conveys.
It takes inputs from the microphone.

2.2.1 Advantages of Proposed System

All-in-one device, where the deaf, dumb and blind can overcome their disabilities

and can express their views to others.

Voice to text Conversion for Deaf People.

Image to Voice Conversion for Blind people.

Sign to text and text to voice conversion for dumb people communicating to normal

person.

It is a portable device with a very low cost.

Chapter 3

LITERATURE SURVEY
“Design of Smart e-Tongue for the Physically Challenged People[16]”

Here they designed a system which converts their sign symbol to text as well as voice
output and normal person’s voice to corresponding sign symbol for two way
communication. This system has flex sensor and IMU (Inertial Measurement Unit) to
recognize their sign symbol, speech synthesis chip for voice output and speech recognizing
module for converting voice to sign symbol. These are interfaced with microcontroller,
which is programmed to obtain corresponding output.

Figure 3.1: Digital display showing Welcome and How are you

“MyVox—Device for the communication between people: blind, deaf, deaf-

blind and unimpaired[17]”
This paper presents the design, prototype and testing of a portable keyboard and speaker
device with a braille refreshable display for the communication between two people (either
being deaf-blind) that has both, a comparatively low cost, and many possibilities for further
development on the ARM-based computer system(Raspberry Pi).

Figure 3.2: model of a system

 “PiCam:IoT based Wireless Alert System for Deaf and Hard of Hearing[18]”
The objective of this paper is to design and implement a low cost stand-alone device for
deaf people to notify doorbell ringing who live alone in their house. The system is based
on Raspberry pi which includes camera, vibrator, wireless GSM and bluetooth. When the
visitor presses the doorbell, captured image is transferred to the wearable device which
helps to know the right person at the door or intruder. After transferring image, wearable
device vibrates to notify. Also, the message is sent to the owner through GSM. Visitor’s
image along with the date and time is sent to the server for retrieving information later. The
system is reliable, effective, and easy to use and also enhances the security of the user.

Figure 3.3: Architectural Diagram using Picam

“Design of a communication aid for physically challenged[19]”

The proposed work in this paper is to implement a system without handheld gloves and
sensors and by capturing the gestures continously and converting them to voice and vice
versa, thus making the communication simpler for deaf and dumb people by a handheld
embedded device along with the hardware setup. The effectiveness of the work was verified
under MATLAB environment and further in future dedicated voice output will be produced
corresponding to the text and the gesture images captured.

Figure 3.4: Text output for Okay and Zero

 “BLIND READER: An Intelligent Assistant for Blind[20]”
The solution is rather simple, introduce a smart device with a multimodal system that can
convert any document to the interpreted form to a blind. A blind can read document only
by tapping words which is then audibly presented through text to speech engine. “Blind
Reader” – developed for touch devices which is user friendly and effective interactive
system for visionless or low vision people.

Figure 3.5: Reconstruction of text in the Interface

“Implementation Of Gesture Based Voice And Language Translator For

Dumb People[21]”
This paper proposes a system that converts gestures given by the user in the form of English
alphabets into corresponding voice and translates this English voice output into any other
Microsoft supported languages. The system consists of MPU6050 for sensing gesture
movement, Raspberry pi for processing, three button Keypad and speaker. It is
implemented by using trajectory recognition algorithm for recognizing alphabets.
Raspberry pi generates voice output for the text in multiple languages using voice RSS and
Microsoft translator. When tested, the system recognized A-Z alphabets and generated
voice output based on the gestures in multiple languages.

Figure 3.6: Experimental Setup and Test mode result

 “Real Time Two Way Communication Approach for Hearing Impaired and
Dumb Person Based on Image Processing[22]”
The objective of this research is to develop a real time system for hand gesture recognition
which recognize hand gestures, features of hands such as peak calculation and angle
calculation and then convert gesture images into voice and vice versa. To implement this
system we use a simple night vision web-cam with 20 megapixel intensity. The ideas
consisted of designing and implement a system using artificial intelligence, image
processing and data mining concepts to take input as hand gestures and generate
recognizable outputs in the form of text and voice with 91% accuracy.

Figure 3.7: Gesture to Speech Conversion

“Smart Cane Location Guide for Blind Using GPS[23]”

In order to make the visually impaired mobility becomes easier, we created a tool that can
help them to detect the location of the destination buildings or inform them where the
position of the standing place through the sound as the output information. Prototype made
using the GPS module, keypad, earphone and raspberry Pi. The results show that the
functional testing tools can detect the location area of interest with a deviation of less than
8 m. Also based on the friendly user testing, this prototype categorized good enough to be
used by the blind.

Figure 3.8: Block Diagram of a System and Testing Prototype to the User
 “Assistive System for Physically Disabled People using Gesture
Recognition[24]”
In this paper we have proposed a method based on hand gesture recognition to recognize
the different gesture used by deaf person to communicate using scale invariant feature
transform (SFIT) algorithm. This will make a bridge between deaf and dumb people and
normal public. In earlier system the uses of colour markers and gloves for gesture
recognition has been used but it resulted in delay in processing time and sometimes
inconvenient for the user. This system will focus on hand gesture recognition and
development of human computer interface (HCI) system which will achieve accuracy, real
time implementation of gesture processing and reduced processing time.

Figure 3.9: The Database Pictures to recognize the Gestures

“A novel approach as an aid for blind, Deaf and dumb people[25]”

In this project, we are going to propose a new system-prototype in an effort to make the
process of interaction between the Blind, Deaf and Dumb people much easier. This will
make use of the Portable Technology and Arduino Circuit Boards to provide a means of
communication to differently-abled people having one or all of the above mention
disabilities.

Figure 3.10: Gesture Pattern, Braille Output, Arduino Circuit and Experimental setup
 “Fitting like a GlovePi: a wearable device for deaf-blind people[26]”
The paper presents the design and implementation of a low cost and open source assistive
system exploiting a wearable device in order to support deaf-blind people in
communication using the Malossi alphabet. More specifically, the system, that we called
GlovePi, is composed by three main low cost components, such as: (i) a gardener glove;
(ii) a Raspberry Pi; (iii) a MPR121 capacitive touch sensor module with expansion board.
The MPR121 module works as bridge between the Raspberry Pi and the sensors in the
glove, allowing the data transfer. Instead, the Raspberry Pi works as a hotspot wifi and as
a server to transfer data to the client, an android application. In this way, the deaf-blind user
can use the glove to deliver messages to other users, using the Malossi alphabet. The
characters (and phrases) in this way created, will be sent to the android application and
displayed or listen.

Figure 3.11: GlovePi Prototype

“Development of Full Duplex Intelligent Communication System for Deaf

and Dumb People[27]”
The paper presents a vision based approach where continuous sequence of frames of
dynamic hand gestures have been taken. Pre-processing, Segmentation, Feature Extraction
and Classification these four are main steps to recognize the dynamic hand gestures. After
that, dual way communication is performed. First the recognized gesture has converted into
text message and voice format, so that the normal person can understand it. Second, voice
has converted into text message and its corresponding gesture, so that physically impaired
humans can understand it. This can break a barrier between deaf-dumb and normal people.

Figure 3.12: Sign Language and Filtered Image

Chapter 4

SYSTEM ANALYSIS AND REQUIREMENTS

4.1 Introduction

Analysis is the process of breaking a complex topic or substance into smaller parts to gain
a better understanding of it. Analysts in the field of engineering look at requirements,
structures, mechanisms, and systems dimensions. Analysis is an exploratory activity.

The Analysis Phase is where the project lifecycle begins. The Analysis Phase is
where you break down the deliverables in the high-level Project Charter into the more
detailed business requirements. The Analysis Phase is also the part of the project where
you identify the overall direction that the project will take through the creation of the project
strategy documents.

Gathering requirements is the main attraction of the Analysis Phase. The process of
gathering requirements is usually more than simply asking the users what they need and
writing their answers down. Depending on the complexity of the application, the process
for gathering requirements has a clearly defined process of its own. This process consists
of a group of repeatable processes that utilize certain techniques to capture, document,
communicate, and manage requirements

4.2 Software Requirement Specification

A Software Requirements Specification (SRS) – a requirements specification for

a software system – is a complete description of the behaviour of a system to be developed.
In addition to a description of the software functions, the SRS also contains non-functional
requirements. Software requirements are a sub-field of software engineering that deals with
the elicitation, analysis, specification, and validation of requirements for software.

4.3 Requirements

4.3.1 Functional Requirement

Device should do minimal computations on its own.

Device should give vocal feedbacks to the user via headphones.
 Device should be able to capture image and display it on the screen meanwhile the
voice should be generated.
Device should be able to convert the received speech-to-text.

4.3.2 Non Functional Requirement

The camera is enabled during the image to speech conversion.

The images will be refreshed as the new image is captured.

4.3.3 Hardware Requirements

HARDWARE:

 System : Pentium IV 2.4 GHz or more

 Hard Disk : 40 GB.
 Monitor : 15 VGA Colour.
 Mouse : Logitech.
 Ram : 512 Mb


4.3.4 Software Requirements

Raspbian OS / Noobs OS
Tesseract OCR
Open CV
Espeak
Xming
Putty
Description about Modules

Tesseract OCR

Python Tesseract is an optical character recognition (OCR) engine for various OS. Tesseract
OCR is the process of electronically extracting1text from images and1reusing it in a variety
of ways1such as document1editing,1free-text1searches. OCR is a technology that is
capable converting documents such as scanned papers, PDF files and captured image into
editable data. Tesseract can be used for Linux, Windows and Mac OS. It can be used by
programmers to extract typed, printed text from images using an API. Tesseract can use
GUI from available 3rd party page. The installation process of tesseract OCR is a
combination of two parts-The engine and training data for a language. For Linux OS,
Tesseract can be obtained directly from many Linux distributers. The latest stable version
of tesseract OCR is 3.05.00. In our project Tesseract is used to convert the captured image
text into text format. Tesseract Features: 1) Page layout analysis. 2) More languages are
supported. 3) Improve forecast accuracy. 4) Add UI.

Figure 4.8: Tesseract OCR

Open CV

It is a library of programming1functions mainly aimed at real-time1computer vision. It is

developed by Intel research center and subsequently supported by1Willow Garage and now
maintained by itseez. It is written in C++ and its primary interface is also in C++. Its
binding is in Python, Java, Matlab. OpenCV runs on a variety of platform i.e. Windows,
Linux, MacOS, openBSD in desktop and Android, IOS and Blackberry in mobile. It is used
in diverse purpose for facial recognition, gesture recognition, object identification, mobile
robotics, segmentation etc. It is a combination of OpenCV C++ API and Python language.
In our project we are using OpenCV version 2 OpenCV is used to gesture control to open
a camera and capture the image. It is also used in the image to text and voice conversion
technique.

Figure 4.9: Open CV

 Espeak

It is a compact open source software speech synthesizer for English and 11other languages
for Linux and Windows platform. It is used to convert text to voice. It supports many
languages in a small size. The programming for espeak software is done using rule files
with feedback. It supports SSML. It can be modified by voice variant. These are text files
which can change1characteristics such as1pitch range, add effects1such as echo, whisper
and croaky voice, or make systematic adjustments to formant frequencies to change the
sound of the voice. The default speaking speed of 180 words per minute is too fast to be
intelligible. In our project Espeak is used to convert the text to voice signal.

Figure 4.10: Espeak

Xming

Xming provides the X Window System display server, a set of traditional sample X
applications and tools, and a set of fonts. It features support of several languages and
has Mesa 3D, OpenGL, and GLX 3D graphics extensions capabilities. The Xming X server
is based on Cygwin/X, the X.Org Server. It is cross-compiled on Linux with
the MinGW compiler suite and the Pthreads-Win32 multi-threading library. Xming runs
natively on Windows and does not need any third-party emulation software. Xming may
be used with implementations of Secure Shell (SSH) to securely forward X11 sessions
from other computers. It supports PuTTY and ssh.exe, and comes with a version of
PuTTY's plink.exe. The Xming project also offers a portable version of PuTTY. When SSH
forwarding is not used, the local file Xn.hosts must be updated with host name or IP address
of the remote machine where GUI application is started. The software has been
recommended by authors of books on free software when a free X server is needed and
described as simple and easier to install though less configurable than other popular free
choices like Cygwin/X.

Figure 4.11: Xming

Putty

Putty is a secluded and open-source mortal emulator, serial comfort network file transfer
application. Putty was formulated for Microsoft Windows, but it has been ported to various
other operating systems. It can link up to a serial port. It backs up a variety of network
protocols, together with SCP, SSH, Telnet, and raw socket connection.

Figure 4.12: Putty

Chapter 5

SYSTEM DESIGN
System design is the process of defining the architecture, components, modules, interfaces
and data for a system to satisfy specified requirements. System design could see it as the
application of systems theory to product development. Theory is some overlap with the
disciplines of system analysis, systems architecture and systems engineering.

If the broader topic development “blends the perspective of marketing, design, and
manufacturing into a single approach to product development,” then design the act of
talking the marketing information and creating the design of the product to be
manufactured. Systems design is therefore the process of defining and developing systems
to satisfy specified requirements of the user.

Until the 1990s systems design had crucial and respected role in the data processing
industry. In the 1990s standardization of hardware and software resulted in the ability to
build modular systems. The increasing importance of software running on generic
platforms has enhanced the discipline of software engineering.

Object-oriented analysis and design methods are becoming the most widely used
methods for computer systems design. The UML has become the standard language in
object-oriented analysis and design. It is widely used for modelling software systems and
is increasingly used for high designing non-software systems and organizations.

System design is one of the most important phases of software development

process. The purpose of the design is to plan the solution of a problem specified by the
requirement documentation. In other words the first step in solution is the design of the
project.

The design of the system is perhaps the most critical factor affecting the quality of
the software. The objective of the design phase is to produce overall design of the software.
It aims to figure out the modules that should be in the system to fulfil all the system
requirements in efficient manner.

The design will contain the specification of all the modules, their interaction with
other modules and the desired output from each module. The output of the design process
is a description of the software architecture.
5.1 High level design

Data flow diagram

A data flow diagram (DFD) is a graphical representation of the flow of the visualization of
data processing. On a DFD, data items flow from an external data source or internal data
source to internal data source or external data sink via an internal process. DFD provides
no information about the timing of process or about whether process will operate in
sequence or in parallel.

Figure 5.2: Dataflow diagram of the system

Chapter 6
SYSTEM IMPLEMENTATION
6.1 Introduction
Implementation is the realization of an application, or execution of a plan, idea, model,
design, specification, standard, algorithm, or policy. In other words, an implementation is
a realization of a technical specification or algorithm as a program, software component,
or other computer system through programming and deployment. Many implementations
may exist for a given specification or standard.

Implementation is one of the most important phases of the Software Development Life
Cycle (SDLC). It encompasses all the processes involved in getting new software or
hardware operating properly in its environment, including installation, configuration,
running, testing, and making necessary changes. Specifically, it involves coding the system
using a particular programming language and transferring the design into an actual working
system.

This phase of the system is conducted with the idea that whatever is designed should be
implemented; keeping in mind that it fulfills user requirements, objective and scope of the
system. The implementation phase produces the solution to the user problem.

6.2 Overview of System Implementation

There could be many ways of implementing the this project, we have chosen Python
because python is a widely used high-level, general-purpose, interpreted, dynamic
programming language. Its design philosophy emphasizes code readability, and its syntax
allows programmers to express concepts in fewer lines of code than would be possible in
languages such as C++ or Java. The language provides constructs intended to enable clear
programs on both a small and large scale. Python supports multiple programming
paradigms, including object-oriented, imperative and functional programming or
procedural styles. It features a dynamic type system and automatic memory management
and has a large and comprehensive standard library. Python interpreters are available for
installation on many operating systems, allowing Python code execution on a wide variety
of systems. Using third-party tools, such as Py2exe or Pyinstaller, Python code can be
packaged into stand-alone executable programs for some of the most popular operating
systems, allowing the distribution of Python-based software for use on those environments
without requiring the installation of a Python interpreter. CPython, the reference
implementation of Python, is free and open-source software and has a community-based
development model, as do nearly all of its alternative implementations. CPython is
managed by the non-profit Python Software Foundation.

6.3 Pseudo code

Pseudo code is an informal high-level description of the operating principle of a computer

program or other algorithm. It uses the structural conventions of a programming language,
but is intended for human reading rather than machine reading. Pseudocode typically omits
details that are not essential for human understanding of the algorithm, such as variable
declarations, system-specific code and some subroutines. The programming language is
augmented with natural language description details, where convenient, or with compact
mathematical notation.

The purpose of using pseudo code is that it is easier for people to understand than
conventional programming language code, and that it is an efficient and environment-
independent description of the key principles of an algorithm. It is commonly used in
textbooks and scientific publications that are documenting various algorithms, and also in
planning of computer program development, for sketching out the structure of the program
before the actual coding takes place.

No standard for pseudo code syntax exists, as a program in pseudo code is not an
executable program. Pseudo code resembles, but should not be confused with skeleton
programs, including dummy code, which can be compiled without errors. Flowcharts and
Unified Modeling Language (UML) charts can be thought of as a graphical alternative to
pseudo code, but are more spacious on paper.

The Project is divided into 4 different modules:

1. Text-to-Speech (TTS)
2. Image-to-Speech using camera (ITSE)
3. Gesture-to-Speech (GTS)
4. Speech-to-Text (STT)
6.4 Module Description
6.4.1 Text-to-speech (TTS) algorithm:-
Step 1: Start

Step 2: Choose option OP1 to convert text to voice.

Step 3: Call the function Text-to-Speech ().

Step 5: Convert text to speech using e-speak synthesizer.

Step 6: Voice is generated.

Step 7: Stop

6.4.2 Image-to-speech using camera (ITSC) algorithm:-

Step 1: Start

Step 2: Choose option OP2 to convert image to speech

Step 3: Call the function Image-to-Speech ().

Step 4: Capture the required image.

Step 5: Convert image to text using Tesseract OCR.

Step 6: Split the text into paragraph.

Step 7: Text is displayed on the screen.

Step 8: Next, call TexttoSpeech () function.

Step 9: Convert text to speech using e-speak synthesizer.

Step 10: Voice is generated.

Step 11: Stop

6.4.3 Gesture-to-speech (GTS) algorithm:-

Step 1: Start

Step 2: Choose the option OP3 to convert gestures-to-text.

Step 3: Capture and read the gesture.

Step 4: Crop the useful portion.

Step 5: Convert RGB image to Grey scale

Step 6: Blur the image using Gaussian Blur algorithm.

Step 7: Pass the processed image to threshold method to get image.

Step 8: Find contours and object of the image.

Step 9: Next, find middle part using convex HULL method.

Step 10: Find the defects and edges of image.

Step 11: Find number of angles < 90 degree.

Step 12: Count the number of angles.

Step 13: The respective text is displayed.

Step 14: Stop

6.4.4 Speech-to-Text (STT) algorithm:-

Step 1: Start

Step 2: Choose the option OP4 for Speech-to-Text conversion.

Step 3: Call the function Speech-to-Text ().

Step 4: Open chromium browser and connect to the website

speechtexter.com

Step 5: Select any universal language.

Step 6: Speak now, by turning on microphone.

Step 7: Check, if the voice is perfect.

7.1: Clear the screen.

7.2: The text is displayed.

Step 8: Recognise it as error and resend the voice, goto step 4.

Step 9: Execute the above steps recursively, until correct output is

obtained.

Step 10: Stop

Chapter 7

TESTING AND EXPERIMENTAL RESULTS

7.1 Testing
Testing is finding out how well something works. In terms of human beings, testing
tells what level of knowledge or skill has been acquired. In computer hardware and software
development, testing is used at key checkpoints in the overall process to determine whether
objectives are being met. For example, in software development, product objectives are
sometimes tested by product user representatives. When the design is complete, coding
follows and the finished code is then tested at the unit or module level by each programmer,
at the component level by the group of programmers involved and at the system level when
all components are combined together. There are several types of testing and few of them
are listed below:

Unit Testing
Integration Testing
Functional Testing
Acceptance Testing

7.1.1 Unit Testing

Unit Testing is a level of software testing where individual units/ components of a

software are tested. The purpose is to validate that each unit of the software performs as
designed. A unit is the smallest testable part of any software. It usually has one or a few
inputs and usually a single output. In procedural programming, a unit may be an individual
program, function, procedure, etc. In object-oriented programming, the smallest unit is a
method, which may belong to a base/ super class, abstract class or derived/ child class.
7.1.2 Integration Testing

An integration test checks how two different components or subsystems interact

with each other. Like a unit test, it generally checks for a specific response to a particular
input. Integration testing takes as its input modules that have been unit tested, groups them
in larger aggregates, applies tests defined in an integration test plan to those aggregates,
and delivers as its output the integrated system ready for system testing.

7.1.3 Functional Testing

A functional test is a form of integration test in which the application is run

"literally". You would have to make sure that an email was actually sent in a functional
test, because it tests your code end to end. It is often considered best practice to write each
type of tests for any given codebase. Unit testing often provides the opportunity to obtain
better "coverage": it's usually possible to supply a unit under test with arguments and/or
an environment which causes all of its potential code paths to be executed. This is
usually not as easy to do with a set of integration or functional tests, but integration and
functional testing provides a measure of assurance that your "units" work together, as they
will be expected to when your application is run in production.

Functional tests are typically written using the WebTest package, which provides
APIs for invoking HTTP(S) requests to your application. We also use py.test and pytest-
cov to provide simple testing and coverage reports. The functional tests used in the project
are mentioned below
Figure 7.3: Functional Testing of Xming
 Figure 7.4: Functional Testing of Putty

7.1.4 Acceptance Testing

Acceptance Testing is often the final step before rolling out the application.
Usually the end users who will be using the applications test the application before
‘accepting’ the application. This type of testing gives the end users the confidence that the
application being delivered to them meets their requirements.

7.2 Test Cases

A test case is a set of conditions or variables under which a tester will determine whether a
system under test satisfies requirements or works correctly. The process of developing test
cases can also help find problems in the requirements or design of an application.

Writing good test cases includes

As far as possible, write test cases in such a way that you test only one thing at a time.
Do not overlap or complicate test cases. Attempt to make your test cases ‘atomic’.

Ensure that all positive scenarios and negative scenarios are covered.
Language:
Write in simple and easy to understand language.
Use active voice: Do this, do that.
Use exact and consistent names (of forms, fields, etc).
Characteristics of a good test case:
Accurate: Exacts the purpose.
Economical: No unnecessary steps or words.
 Traceable: Capable of being traced to requirements.
Repeatable: Can be used to perform the test over and over.
Reusable: Can be reused if necessary.

We aim for developing the prototype model for blind dumb and deaf people by
employing in a single compact device. The device provides a unique solution for these
people to manage their sites by themselves. The project is concerned with the source code
of Python. It is the easiest programming language to interface with the Raspberry Pi. The
system is provided with 4 Options. Each option has different functions. We have chosen
the options for necessary conversion.
1) Text to speech (TTS) using (option1)
2) Image to speech using camera (ITSC) using (option2)
3) Gesture control using (option3)
4) Speech to text(STT) using (option4)

Figure 7.5:Test Cases

7.2.1 Text-to-speech (TTS):-
The first process text to speech conversion is done for the dumb masses who cannot speak.

The Dumb people convert their thoughts to text which could be transferred to a voice signal.

The converted voice signal is spoken out by espeak synthesizer. After selecting the option

OP1 the OS and sub process imported. Call text to speech function and enter the text as

input. After entering the text from keyboard, the espeak synthesizer converts text to speech.

The process also provided with the keyboard interrupt ctrl+C.

Figure 7.8: Text-to-Speech

7.2.2 Image-to-speech using camera (ITSC):-
The second process is developed for blind people who cannot read normal text. In order to

help blind people, we have interfaced the Logitech camera to capture the image by using

OPENCV tool. The captured image is converted to text using Tesseract OCR and save the

text to file out.txt. Open the text file and split the paragraph into sentences and save it. In

OCR, the adaptive thresholding techniques are used to change the image into binary images

and they are transferred to character outlines. The converted text is read out by the espeak.

Figure 7.9: Image-to-Speech

7.2.3 Gesture-to-speech (GTS):-
The third process is developed for the vocally impaired people who cannot exchange the

thoughts to the normal people. Dumb people uses gesture to communicate with normal

people which are majorly not understandable by normal people. The process starts with the

capturing of image and crops the useful portion. Convert the RGB image into gray scale

image for better functioning, Blur the cropped image through Gaussian blur function and

pass it to the threshold function to get the highlighted part of the image. Find the contours

and an angle between two fingers. By using convex hull function, we can implement the

finger point. Count the number of angles which is less than 90 degree which gives the

number of defects. According to the number of defects, the text is printed on display and

read out by the Speaker.

Figure 7.10: Gesture-to-text

7.2.4 Speech-to-Text (STT):-
The fourth process is developed for the hearing impairment, people who cannot understand

the words of normal people. In order to help them, our project is provided with a switch

which is used to convert the voice of the normal people text. We have used a chromium

browser which is automatically connected to URL speechtexter.com. The process is

performed by assigning a minimum threshold voltage to recognize the voice signal. The

input is given through a microphone which is converted into a text format. The URL

supports a variety of languages. If the voice signal recognizable it will print the text else it

gives the error signal.

Figure 7.11: Speech-to-Text

Chapter 8

CONCLUSION AND FUTURE ENHANCEMENT

8.1 Conclusion

This project aims to lower the communication gap between the deaf or mute community
and the normal world, help them to lead standard lifestyle. The device is used to convert
text/image to voice for blind, speech to text conversion for deaf and conversion of hand
gestures to text (kannada words) for dumb people. We have designed the prototype model
for blind, deaf and dumb people into a single compact device. The advantage of this device
is that it can be easily carried (portable) due to its less weight and size. The device can be
used as smart assistant for differently abled people to communicate with others and it is a
language independent system.

8.2 Future Enhancement

There can be number of future advancements that can be associated with this project work
and some of which are described as follows:

The system can be further expanded for the alphabets, numbers in gesture control.
The input can be also taken in the form of videos and they are divided into frames
and then it is converted into text.
We can also add grammatical structure for sign language.
The system can be made handy by incorporating it into a mobile phone.
We can produce a product for blind people that converts the information in any
hand-written notes, newspaper or books into an audio signal that these people can
here.
System can be more efficient for all languages.
 BIBLIOGRAPHY
1. Shraddha R. Ghorpade, Surendra K. Waghamare, “Full Duplex Communication
System for Deaf & Dumb People,” International Journal of Emerging Technology
and Advanced Engineering (IJETAE), Volume 5, Issue 5, May 2015, ISSN 2250-
2459.

2. Chucai Yi, Student Member, IEEE, Yingli Tian, Senior Member, IEEE, and
Aries Arditi “Portable Camera-Based Assistive Text and Product Label Reading
From Hand-Held Objects for Blind Persons” 2013 IEEE.

3. Vasanthi.G and Ramesh Babu.Y Department of ECE, DMI College of Engineering,

Chennai, India.“Vision Based Assistive System for Label Detection with Voice
Output”Jan-2014.

4. S. Hariharan,K.Abdul Rahim,T.Ajith Prakash G. Ganesh Kumar,” A New Concept

Of Electronic Braille display Interfaced With Computer” in April-2009.

5. Bachar Y.R, Gupta. R, Pathan W.A (E&T Dept. SIER NASIK, SPP University,
Pune, India)” Smart Speaking Gloves for Speechless “.

6. Dharanikumar Chowdary. M, M. Narasimhulu, G. Subrahmanya Sharma,

“Advanced Embedded Chatter Box for Physically Challenging Persons” in Jan
2012.

7. Bhavina Patel,Vandana Shah, Ravindra k Shirsagar, “Microcontroller Based

gesture Recoginition Syatem For the Handicap People”in Dec-2011.

8. Pallavi Varma, Shimi.S.L, Richapriyadarshini, ”Design Of Communication

Interpreter For Deaf and Dumb person” in Jan-2015.

9. S.B.Shroke, Nandar Dehpande, Prashant Deshmukh, Sanjay Kumar Mathapati. ”

Assistive Transulator For Deaf And Dumb People”-in July-2014.
10. Michael McEnancy” Finger Reader Is audio reading gadget for Index Finger “in
July 2014.

11. M. Delliraj, S.Vijayakumar # Department of Tifaccore in Pervasive computing

Technologies Velammal Engineering College, Chennai, India.2013 International
Conference on Recent Trends in Information Technology (ICRTIT).

12. Fernando Ramirez-Garibay, Cesar Millan Olivarria, Alejandro Federico Eufracio

Aguilera,& Joel C. Huegel Biomechatronics Laboratory ©2014 IEEE.

13. Pushpanjali Kumari, Pratibha Goel, Dr. S. R. N. Reddy, 2015 International

Conference on Advanced Computing and Communications.

14. Suganya R, Dr.T.Meeradevi, IEEE sponsored 2nd international conference on

electronics and communicateon system (icecs 2015).

15. Shahed Anzarus Sabab, Md. Hamjajul Ashmafee, 19th International Conference
on Computer and Information Technology, December 18-20, 2016, North South
University, Dhaka, Bangladesh.

16. L. Anusha, Y. Usha Devi, Vignan’s Lara Institute of Technology and Science,
Implementation Of Gesture Based Voice And Language Translator For Dumb
People,(2016).

17. Shweta S. Shinde, Rajesh M. Autee, Vitthal K. Bhosale, 2016 IEEE International
Conference on Computational Intelligence and Computing Research.

18. Dr P Subashini, Dr M Krishnaveni, M Manjutha, 2Department of Computer

Science, 2016 International Conference on Computational Systems and
Information Systems for Sustainable Solutions.

19. Gita Indah Hapsari, Giva Andriana Mutiara, Dicky Tiara Kusumah, 2017 Fifth
International Conference on Information and Communication Technology
(ICoICT).
20. Subhankar Chattoraj, Karan Vishwakarma, Tanmay Paul, 2017 IEEE 2nd
International Conference on Signal and Image Processing.

21. Rajapandian B, Harini V, Raksha D, Sangeetha V, 2017 IEEE 3rd International

Conference on Sensing, Signal Processing and Security (ICSSS).

22. Silvia Mirri, Catia Prandi, Paola Salomoni, Lorenzo Monti, 2017 14th IEEE
Annual Consumer Communications & Networking Conference (CCNC).

23. Surbhi Rathi, Ujwalla Gawande, Development of Full Duplex Intelligent

Communication System for Deaf and Dumb People, 978-1-5090-3519-
9/17/$31.00_c 2017 IEEE

24. Anchal Sood, Anju Mishra, AAWAAZ: A Communication System for Deaf and
Dumb, 2016 5th International Conference on Reliability, Infocom Technologies
and Optimization (ICRITO) (Trends and Future Directions), Sep. 7-9, 2016,
AIIT, Amity University Uttar Pradesh, Noida, India.
 APPENDIX
Prof. R Chandramma, Lekhana M, Deepa B N, Archana Kumari Jha “A Literature
Survey on Raspberry Pi Based Assistive Communication system for Deaf,
Dumb and Blind”, paper presented in 10th National Conference on Advances in
Information Technology at SJBIT, Bangalore on 9 th May 2018.

Prof. R Chandramma, Lekhana M, Deepa B N, Archana Kumari Jha “Raspberry

Pi Based Communication System for Deaf, Dumb and Blind Person”, paper
presented and applied for journal Publication in National Conference on Science,
Engineering and Management(NCSEM) at Oxford, Bangalore on 24th – 25th May
2018.
Raspberry Pi Based Communication System for Deaf Dumb and Blind Person

Page 55