University Of Sharjah

Electrical and Computer Engineering Department

Electrical and Electronics Engineering
Spring 2016
Senior Design Project 2

Smart Wearable Obstacle Detection

Glasses for Low-vision People
Project Group: Project Examination Committee:
Hamda AlAhmed U00029871 Supervisor: Prof. Soliman Mahmoud
Hamda AlJassmi U00036427 Examiner: Dr. Amer Ghais
Noora Jamal Ahli U00038860 Chairman: Dr. Qassim Nassir

1
‫ ‬
‫ ‬
‫ ‬
‫ﻣﻠﺨﺺ اﻟﺮﺳﺎﻟﺔ‬
‫ ‬

‫ﺗﺸﻴﺮ اﻹﺣﺼﺎﺋﻴﺎت إﻟﻰ أن ﻋﺪد اﻟﻤﻨﺪرﺟﻴﻦ رﺳﻤﻴﴼ ﺗﺤﺖ ﺗﺼﻨﻴﻒ ذوي اﻹﻋﺎﻗﺔ اﻟﺒﺼﺮﻳﺔ ﻳﺼﻞ ﺗﻘﺮﻳﺒﴼ‬
‫إﻟﻰ ‪258‬ﻣﻠﻴﻮن ﺣﺎﻟﺔ ﻓﻲ اﻟﻌﺎﻟﻢ‪39 ،‬ﻣﻠﻴﻮن ﻣﻨﻬﻢ ﻓﺎﻗﺪو اﻟﺒﺼﺮ ﺗﻤﺎﻣﴼ ﺑﻴﻨﻤﺎ ‪248‬ﻣﻠﻴﻮن ﻣﻨﻬﻢ‬
‫ﻳﻤﺘﻠﻜﻮن ﻧﺴﺒﺔ ﺑﺼﺮ ﺿﺌﻴﻠﺔ ﺟﺪﴽ‪ .‬ﺑﺎﻟﺘﻤﻌﻦ ﻓﻲ اﻷرﻗﺎم و ﻓﻲ اﻟﺤﺎﻻت اﻟﻤﺮﺿﻴﺔ ﻣﻦ ﺣﻮﻟﻨﺎ ﻧﺠﺪ أن‬
‫اﻟﻐﺎﻟﺒﻴﺔ اﻟﻌﻈﻤﻰ ﻫﻢ ﻣﻦ أﺻﺤﺎب اﻟﻨﻈﺮ اﻟﻀﻌﻴﻒ‪ ،‬اﻟﺬﻳﻦ ﻗﺪ ﻳﻜﻮﻧﻮ ﻗﻠﻴﻠﻲ ﺣﻆ ﺑﺴﺒﺐ اﻟﺪﻣﺞ ﻏﻴﺮ‬
‫اﻟﻤﻨﺼﻒ ﻓﻲ اﻟﻤﺴﻤﻴﺎت ﻧﻈﺮﴽ ﻹﻣﻜﺎﻧﻴﺔ ﺗﻮﻓﻴﺮ ﺣﻠﻮل واﺳﻌﺔ ﻟﻤﺸﻜﻠﺘﻬﻢ اﻟﺘﻲ ﺗﺨﺘﻠﻒ ﺗﻤﺎﻣﺎ ﻋﻦ‬
‫ﻓﺎﻗﺪي اﻟﺒﺼﺮ و ﺑﺎﻟﺘﺎﻟﻲ ﺗﻮﻓﻴﺮ ﻧﻤﻂ ﻣﻦ اﻟﺤﻴﺎة أﺳﻬﻞ ﺑﻜﺜﻴﺮ ﻳﺨﺘﻠﻒ ﺗﻤﺎﻣﴼ ﻋﻤﺎ ﻫﻢ ﻋﻠﻴﻪ‪ ،‬ﻣﻤﺎ‬
‫ﻳﺴﺎﻫﻢ ﻓﻲ ﺗﻌﺰﻳﺰ ﺛﻘﺘﻬﻢ ﺑﺎﻟﻨﻔﺲ و اﻹﺣﺴﺎس ﺑﺤﻴﺎة ﻧﻮﻋﴼ ﻣﺎ أﻗﺮب ﻟﻠﻄﺒﻴﻌﻴﺔ‪ .‬ﻋﺪة ﺣﻠﻮل ﻇﻬﺮت و‬
‫ﺑﺸﻜﻞ ﻣﺨﺘﻠﻒ ﺗﻌﺎﻟﺞ ﺷﺘﻰ ﻣﺸﻜﻼت اﻟﺘﺄﻗﻠﻢ‪ ،‬و ﻟﻜﻦ ﺣﺘﻰ اﻵن ﻻ ﻳﻮﺟﺪ ﺣﻞ ﺟﺬري ﻣﻠﻤﻮس‪.‬‬
‫ﻳﻬﺪف ﺗﺼﻤﻴﻢ ‪ NOOR‬إﻟﻰ ﺗﺴﻬﻴﻞ ﺣﻴﺎة اﻟﻤﺴﺘﺨﺪم و زﻳﺎدة ﻛﻔﺎءﺗﻪ‪ ،‬دﻗﺘﻪ وﺳﺮﻋﺔ ﺣﺮﻛﺘﻪ ﺣﻴﻦ‬
‫ﻳﻨﺘﻘﻞ ﻓﻲ ﺑﻴﺌﺔ ﻣﻌﻴﻨﺔ ﻣﺠﻬﻮﻟﺔ اﻟﻤﻌﺎﻟﻢ ﺑﺎﻟﻨﺴﺒﺔ ﻟﻪ‪ ،‬ﻋﻦ ﻃﺮﻳﻖ اﺳﺘﺨﺪام ﻛﺎﻣﻴﺮا ﻣﺜﺒﺘﺔ ﻋﻠﻰ ﻫﻴﻜﻞ‬
‫اﻟﻨﻈﺎرة ﺗﻘﻮم ﺑﺘﺼﻮﻳﺮ اﻟﻤﺸﺎﻫﺪ ﺛﻢ ﻣﻌﺎﻟﺠﺘﻬﺎ ﻟﺘﻨﺎﺳﺐ اﻟﻤﺴﺘﺨﺪم و أﺧﻴﺮﴽ إرﺳﺎﻟﻬﺎ إﻟﻰ ﺷﺎﺷﺔ ﺗﻘﻊ‬
‫ﻋﻠﻰ ﻣﻘﺮﺑﺔ ﻣﻦ ﻋﻴﻦ اﻟﻤﺴﺘﺨﺪم‪ ،‬و ﺑﺬﻟﻚ ﺳﺘُﺘﺎح ﻟﺬوي اﻟﺒﺼﺮ اﻟﻤﺤﺪود ﺟﺪﴽ ﻓﺮﺻﺔ اﻟﺘﻌﺮف ﻋﻠﻰ‬
‫اﻟﻌﺎﻟﻢ ﻋﻦ ﻗﺮب ﺑﺸﻜﻞ أﻛﺒﺮ‪ .‬أﺛﺒﺘﺖ ﻧﺘﺎﺋﺞ اﻻﺧﺘﺒﺎرات اﻟﺘﻲ أﻗﻴﻤﺖ ﻋﻠﻰ ذوي اﻟﻨﻈﺮ اﻟﻤﺤﺪود أن‬
‫اﻻﺳﺘﺨﺪام اﻟﻤﺘﻜﺮر ل ‪NOOR‬و ﻣﻊ اﻟﺘﻌﻮد ﻋﻠﻴﻬﺎ ﻳﺴﺎﻋﺪ اﻟﻤﺴﺘﺨﺪم ﻓﻲ اﺟﺘﻴﺎز اﻷﺷﻴﺎء ﻣﻦ ﺣﻮﻟﻪ‬
‫دون اﻻﺻﻄﺪام ﺑﻬﺎ و ﺑﺎﻟﺘﺎﻟﻲ ﻳﻌﺰز ﻟﺪﻳﻪ اﻟﻘﺪرة ﻋﻠﻰ اﻻﻋﺘﻤﺎد ﻋﻠﻰ ﻧﻔﺴﻪ ﻓﻲ اﻟﺘﻨﻘﻞ ﻣﻦ ﻣﻜﺎن‬
‫ﻵﺧﺮ‪.‬‬
‫ ‬
‫ ‬
‫ ‬ ‫ ‬

‫ ‬ ‫ ‪2‬‬
 Abstract


Estimated statistics show that 285 million people around the world are considered
blind; 246 million of them remain very low vision while 39 are totally blind, and as noticed,
the majority of them are with low-vision. People under this classification may be oppressed,
knowing that they have a better chance to improve their quality of life, and therefore,
provide a better lifestyle which contributes to build up a stronger self-confidence, while the
totally blind people have much narrowed options. Many researches with different solutions
have been done, but yet, no radical solution can cure and fix the various cases. The design
of the proposed smart glass aims to facilitate the users’ life and increase their efficiency,
reliability, and mobility while moving through an unknown environment. This is done by
placing a stereo-vision camera on the body of the glass that captures the surrounding
environment, processes it, and sends it to a screen that is close to the patient’s eyes. By
that, they will have the opportunity to experience the world in an exceptional way.
Participants were able to walk through an unknown environment with different set-up
mazes independently, where the results showed that the time taken to complete a set-up
maze decreased logarithmically, number of obstacle collisions decreased linearly, and the
user's velocity increased linearly.

3
 Table Of Content

Chapter 1 ............................................................................................................................................................... 9
1 Introduction ................................................................................................................................................. 9
1.1 Background .......................................................................................................................................... 9
1.2 Cases ....................................................................................................................................................... 9
1.3 Thesis objective ............................................................................................................................... 10
1.4 Thesis organization ........................................................................................................................ 11
Chapter 2 ............................................................................................................................................................. 12
2 Terminology ............................................................................................................................................... 12
2.1 Introduction ...................................................................................................................................... 12
2.2 Technical terminology .................................................................................................................. 12
2.2.1 Stereovision ................................................................................................................................ 12
2.2.2 Depth map .................................................................................................................................. 13
2.2.3 Back-illumination sensor ........................................................................................................ 14
2.3 Medical terminology ...................................................................................................................... 15
2.3.1 Retinitis Pegmentosa (RP) ..................................................................................................... 15
2.3.2 Glaucoma .................................................................................................................................... 16
2.3.3 Cataracts ...................................................................................................................................... 17
2.3.4 Diabetic Retinopathy ............................................................................................................... 18
2.3.5 Age-related macular degeneration .................................................................................... 19
Chapter 3 ............................................................................................................................................................. 21
3 Related work .............................................................................................................................................. 21
3.1 Introduction ...................................................................................................................................... 21
3.2 Ultrasonic sensor ............................................................................................................................ 21
3.3 Infrared distance sensor .............................................................................................................. 22
3.4 Assistance aids ................................................................................................................................. 22
3.4.1 Smart cane .................................................................................................................................. 23
Figure 3.2 : The Blindsport cane structure [22] ............................................................................ 23
3.4.2 Applications with sensors ...................................................................................................... 23
3.5 Technical designs in the markets ............................................................................................. 24
3.5.1 Google glass ................................................................................................................................ 24
3.5.2 OrCam ........................................................................................................................................... 25
3.5.3 Depth based smart glasses .................................................................................................... 26
Chapter 4 ............................................................................................................................................................. 28
4 Synthesis ...................................................................................................................................................... 28
4.1 Introduction ...................................................................................................................................... 28
4.2 Depth cameras .................................................................................................................................. 28
4.2.1 Orbbec .......................................................................................................................................... 29
4.2.2 Microsoft Kinect ........................................................................................................................ 31
4.2.3 Asus Xtion Pro ............................................................................................................................ 33
4.2.4 ZED ................................................................................................................................................. 34
4.2.5 Alternatives comparison ........................................................................................................ 37
4.3 Jetson TK1 .......................................................................................................................................... 37
4.4 Displays ............................................................................................................................................... 39
4.4.1 OLED .............................................................................................................................................. 39
4.4.2 TFT ................................................................................................................................................. 40

4
 Choice 2: ..................................................................................................................................................... 41
4.5 Battery ................................................................................................................................................. 42
4.6 Conclusion .......................................................................................................................................... 42
Chapter 5 ............................................................................................................................................................. 43
5 Proposed design ....................................................................................................................................... 43
5.1 Proposed solution ........................................................................................................................... 43
5.2 Hardware design ............................................................................................................................. 44
5.3 Software design ............................................................................................................................... 45
5.3.1 ZED SDK ........................................................................................................................................ 46
5.4 Testing methods .............................................................................................................................. 53
5.4.1 Study 1: Obstacle avoidance test on healthy people with normal vision ............. 53
5.4.2 Study 2: Obstacle avoidance test on low-vision people .............................................. 56
Chapter 6 ............................................................................................................................................................. 58
6 Constraints and specifications ........................................................................................................... 58
6.1 Introduction ...................................................................................................................................... 58
6.2 List of constraints ........................................................................................................................... 58
6.2.1 Environmental concerns ........................................................................................................ 58
6.2.2 Ethical and social issues .......................................................................................................... 58
6.2.3 Economic issues ........................................................................................................................ 58
6.2.4 Limitations and restrictions .................................................................................................. 59
6.3 List of specifications ...................................................................................................................... 59
6.3.1 Parts list and estimated budget ........................................................................................... 59
4755 ...................................................................................................................................................................... 59
Chapter 7 ............................................................................................................................................................. 61
7 Conclusion .................................................................................................................................................. 61
7.1 Thesis summary .............................................................................................................................. 61
7.2 Difficulties .......................................................................................................................................... 61
7.3 Future work ....................................................................................................................................... 62

5


List of Tables

Table 4.1: The specifications of Orbbec Persee ...................................................................... 30
Table 4.2: Orbbec Astro Pro Specifications ............................................................................ 30
Table 4.3: Microsoft kinect specifications. ............................................................................. 32
Table 4.4: Asus Xtion specifications ........................................................................................ 33
Table 4.5: The output resolution of ZED ................................................................................. 34
Table 4.6: Physical specifications of ZED ................................................................................ 35
Table 4.7: Alternatives main differences ................................................................................ 37
Table 4.8: Specifications of Jetson TK1 .................................................................................. 38
Table 4.9: 1.66" flexible OLED specifications .......................................................................... 40
Table 4.10: TFT LCD model technical specifications [42] ........................................................ 40
Table 4.11: Adafruit display specifications ............................................................................. 42
Table 5.1: Key shortcuts to control different characteristics of depth image ........................ 48
Table 5.2: Low-vision participants details .............................................................................. 56
Table 6.1: Parts List and Estimated Budget ............................................................................ 59

6

List of figures

Figure 1.1: the block diagram of the entire system design .................................................... 10
Figure 2.1: Disparity map generation . ................................................................................... 13
Figure 2.2: Triangulation diagram .......................................................................................... 14
Figure 2.3: Normal Retina (Left) vs. Retinitis Pegmentosa (Right) ......................................... 15
Figure 2.4: The developing stage of the vision for a person with glaucoma ......................... 17
Figure 2.5: blurry and indistict image results while having a cataract .................................. 18
Figure 2.6: A person with Diabetic Retinopathy will see blurry and distorted images (left) . 19
Figure 2.7: AMD cause a blind spots and blurry image (Right) .............................................. 20
Figure 3.1: Seeedstudio Ultrasonic ........................................................................................ 21
Figure 3.2 : The Blindsport cane structure ............................................................................. 23
Figure 3.3: Prototype of gloves with sensors ......................................................................... 24
Figure 3.4: Google glass .......................................................................................................... 25
Figure 3.5: OrCam ................................................................................................................... 25
Figure 3.6 : A) Prof. Stephen Hicks mart glasses with a laptop that should be carried all the
time. B) Different mazes of a single corridor for tests purposes. C) Time, velocity and
total collisions vs. number of trials graphs. .................................................................... 27
Figure 4.1: Orbbic Persee ....................................................................................................... 29
Figure 4.2: Orbbec Astra Pro .................................................................................................. 30
Figure 4.3: A human standing at 6m distance can be clearly distinguished in the ................. 31
Figure 4.4: Microsoft Kinect sensor with the cover taken off ............................................... 31
Figure 4.5: Microsoft Kinect sensor depth image .................................................................. 32
Figure 4.6: Human face captured with Microsoft Kinect sensor. (a) Video frame, (b) depth
image, and (c) a close-up of the facial surface .............................................................. 32
Figure 4.7: Asus Xtion ............................................................................................................ 33
Figure 4.8: ZED stereovision camera ...................................................................................... 34
Figure 4.9: original image and the depth map corresponding to it using ZED SDK with
OpenCV library ............................................................................................................... 35
Figure 4.10:How a ZED camera works .................................................................................... 36
Figure 4.11: A block diagram of using ZED to generate a depth map .................................... 36
Figure 4.12: Jetson TK1 ........................................................................................................... 38
Figure 4.13L Flexible OLED .................................................................................................... 39
Figure 4.14: TFT LCD model .................................................................................................... 41
Figure 4.15: Adafruit 5" display .............................................................................................. 41
Figure 5.1: Hardware structure of NOOR ............................................................................... 44
Figure 5.2: The block diagram of the entire system design .................................................... 45
Figure 5.3: OpenCV codes flowchart ...................................................................................... 47
Figure 5.4: A combination of the left and right images, default in OpenCV ........................... 48
Figure 5.5: Left view (on the left), and right view (on the right) ............................................ 48
Figure 5.6: Overlay image of left and right sources ................................................................ 49
Figure 5.7: Depth map ............................................................................................................ 49
Figure 5.8: FULL sensing mode (right), RAW sensing mode (left) ........................................... 49
Figure 5.9: Performance mode (left), Quality mode (right) .................................................... 50

7
Figure 5.10: Left and right views in disparity .......................................................................... 50
Figure 5.11: Decrease (left) and increase (right) reliability index by 1 in RAW mode ........... 50
Figure 5.12:Decrease (left) and increase (right) reliability index by 1 in FULL mode ............. 51
Figure 5.13: depth map represented by OpenCV. .................................................................. 51
Figure 5.14: depth map represented by OpenGL ................................................................... 52
Figure 5.15: OpenGL code flowchart ...................................................................................... 52
Figure 5.16: A photo of one arrangement of the different obstacles and schematics show the
setup up of the five mazes ............................................................................................. 54
Figure 5.17: a) Number of trials vs. Collisions graph. b) Number of trials vs. Time graph. c)
Number of trials vs. Velocity graph, which belongs to the average results . .................. 55
Figure 5.18: a) Number of trials vs. Collisions graph. b) Number of trials vs. Time graph. c)
Number of trials vs. Velocity graph, which belongs to participant 2 .............................. 57

8
 Chapter 1

1 Introduction
1.1 Background
According to the World Health Organization, there are around 285 million
people in the world considered blind, the majority (246 million) has low-vision and the rest
are totally blind. The statistics also mention that 82% of the blind people are above 50, and
the numbers of the visual impairments has decreased upon the last 20 years according to
the different and successful treatment methods [1].

Scientists and doctors classify the vision level into 4 important categories: the
normal vision, middle visual impairments, severe visual impairments which identify the term
“low-vision”, and finally the blindness.

During the research the focus will be on the low-vision category since they are the
majority and a lot of help can be given to them so they can have a better quality of life.
Since that no patients are identical; five main cases that lead to vision loss will be discussed
which are: Retinitis Pigmentosa, Glaucoma, Cataracts, Diabetic Retinopathy, and Age-
Related Macular degeneration [2].

A patient who’s under these categories may be able to get benefit from the design.
The focus here will be on the symptoms, once they are known, designing the glasses can be
done to enhance and improve the quality of the reflected images on their eyes in a way that
suits their conditions, but not treating them.

This chapter is organized as follows: section 1.2 briefly explains the five main cases.
Section 1.3 illustrates the main objective of the project and shows the block diagram.
Section 1.4 mentions the thesis organization.

1.2 Cases
This project focuses on five intrinsic cases for the purpose of helping patients with
these cases, however the tests were done on two cases only. The cases are summarized as
follows:

• Retinitis pigmentosa, where the patient gradually lose the night and
circumferential vision, which leads up to the tunnel vision and blurry view.
• Glaucoma, patients with this disease face an unexpected sight vision loss with hazy
view and acute head pains due to the altitude compression on the eye.

9
 1.3 Thesis objective
The intention of the project is to resolve the problem of vision with the low-vision
people, who are officially classified as blind persons. Although they are considered blind,
they have different issues that can be handled with variety of solutions; one of them is to
improve the remained vision by enhancing various environment scenes. This can be
achieved by implementing a smart glass –NOOR- that depends on capturing the scenes,
transforming the scene to a depth map, and after all, reflecting the new scenes on a screen.

The significant value of the project is the improvements in the wearer’s social and
academic life. Identically, wearing NOOR helps the user to avoid the surrounding diversified
obstacles and thus the mobility increases. If such a product is released, the independency of
the users can be augmented, which will get them involved more in the society. The project’s
strategy is to use the product on people with different low-vision cases, where the low-
vision patients can communicate better with others, reach desired destinations faster and
easily, and participate in the various events. Moreover, disposing of canes make a person
looks normal.

In order for NOOR to function well, a special camera mounted on the top of the
glasses’ body reconstructs a depth map scene and a display role as the lenses of the glass
receive the reconstructed scene’s video signal with a frame rate of at least 30 fps and
presents it.

Figure 1.1 shows the block diagram of the entire system design. By the time the
camera detects the surrounding scenes, it conveys it to an integrated graphic processor unit
(GPU) to enhance the images with the help of Jetson TK1, henceforth a depth map of the
original image is generated which will be delivered to the display to fit the different
conditions.

Jetson TK1

Left
rd
source ZED SDK + 3 party library
Depth
Image Image 3D map
Detection map
Camera (Video
analysis Correction generation
stream) Right
source

Display

Figure 1.1: the block diagram of the entire system design

10
1.4 Thesis organization
The thesis is organized as follows: chapter 2, which comes after the introduction
represents some technical and medical terminologies in order to simplify the thesis for the
reader. Chapter 3 discusses the previous works done that are related to the proposed
design, chapter 4 compares between different cameras and displays and shows the
specifications of each and mentions the chosen components, chapter 5 illustrates the
proposed design, explains the final block diagram, presents tests methodologies and shows
all the results, chapter 6 lists the constraints and specifications of the system, and chapter 7
concludes the work that has been done.

11
 Chapter 2

2 Terminology
2.1 Introduction
In order to simplify the important concepts of the proposed design, some of the
technical terms, methods, and medical cases should be cleared, therefore, this chapter goes
through all the techniques and the terminologies in a simple and useful way while
explaining them briefly. It is arranged as follows: section 2.2 mentions the technical
terminologies used in the thesis. Section 2.3 demonstrates the five main disease cases and
explains their symptoms and treatments.

2.2 Technical terminology

2.2.1 Stereovision

If we have two cameras calibrated and placed side-by-side then we can obtain 3D
information from an arbitrary scene and the position of the scene’s points, this process is
called the stereovision. It is defined as the way of getting depth information about a certain
scene from two 2D images. Provided that this technique is taken from the human and
animals eyes, and it has been applied to the computers to give the devices a new vision [3].

Left and right images are taking from two cameras and are reconstructed to create
the geometry of the scene; usually there is a horizontal shift between the two images.

2.2.1.1 Stereo principle

In order to create a stereovision, two cameras are placed side-by-side horizontally to

gain two different views of the same scene, and then these two images are being compared
so the depth information can be constructed in a disparities form, which is inversely
proportional to the differences in objects’ distances [4].

If the intrinsic parameters of each camera and the relative pose between the
cameras are known, and if an image point in the left camera and its corresponding point in
the right camera are measured, the absolute point position can be found by intersecting
(triangulate) the rays.

12
 2.2.2 Depth map

The least difficult and most helpful method of performing depth estimations taken
from an image is the depth map. It is a 2D array where the distance information is
represented by the regular x and y axes and the compared z measurements are stored in
the pixels. Depth map contains the information that is related to the distance and the exact
location of an object or a surface on a scene. It is a 3D image that consists of gray pixels,
with the values from 0 to 255; the near the object is, the lighter the represented color,
therefore, the nearest object represents the “0” value, while the far objects is represented
by darker colors which corresponds to high values [5].

The generated depth map from the stereovision camera is supported by the
triangulation method. As known, it has two stereo cameras placed one next to the other
horizontally with a certain distance between them. The scenes captured from these
cameras have different angles of view, and the cameras have been utilized and organized in
a way that they overlap at the required distance of the object. The result is two different
viewpoints for each object in the scene [6].

Two actions should be taken into considerations in order to reach the goal of
obtaining a 3D depth map from a scene: First, analyzing the corresponding left and right
images’ point for a specific surface point that appears in the scene is a must. Secondly, to
calculate the ray intersection point of the pixels in the left and right cameras, the structure
and the geometry of the camera should be known [7].

The used technique is quite simple, the disparity map which is the final look of the
depth map, consists of many disparity values, each value has been calculated using the
difference of the left and right pixels of the same point, figure 2.1 illustrates this clearly.

Figure 2.1: Disparity map generation [7].

13
 The mathematical equation (1) is needed to calculate the depth corresponding value
for each point of the scene. And the equivalent triangle diagram for equation (1) is
represented in figure 2.2.

! !∗!
Disparity = X − X = ! (1)

Figure 2.2: Triangulation diagram [8]

Mentioning that B is the separation between the two cameras, x and x' are the
separation between points in picture plane relating to the 3D scene point and their
camera's center. And f is the central length of camera. To sum up with, the depth of the
image's pixels can be obtained if B, x-x’, and f were given [8].

2.2.3 Back-illumination sensor

Back-illumination sensor is defined as a kind of advanced image sensor that uses a

novel plan of the imaging components to raise the number of light caught and in this
manner to enhance the performance in low-light environments, it is also named as backside
illumination (BSI) [9]. Such sensors can be used for many applications like security cameras
as well as in smart phones, and it is considered as the future of digital imaging and
photography because of the high performance it can reach.

The front-illumination, customary approach to sense the light, works like the human
eyes' rule. The photo detectors in this situation lies behind the lens where lattice that
contains the photo components and wiring has been submitted in front in order to simplify
the assembling, however the wiring is keeping the photocathode layer from accepting all
the light data and in this case diminishing the sign that is accessible to be caught.

14
 The BSI sensor has this noteworthy productivity on the grounds because the light
encroaches straightforwardly onto the photoreceptors, instead of threading its way through
a maze of hardware as on an ordinary sensor. This is accomplished by turning the sensor
over so that the inverse side of the sensor gathers the light, henceforth the term 'posterior
brightening'. As for gathering photons, is around half more than the normal customary
sensor [10].

2.3 Medical terminology

2.3.1 Retinitis Pegmentosa (RP)

There are more than 30 million people with Retinitis Pegmentosa around the world,
and yet, there is only one treatment that affects only 10 percent of the disease. Therefore,
the unmet need is major, especially for people with advance RP. Nowadays, many people
suffer with this disease; unfortunately, medical communities believe that there is nothing
that can be done to treat RP.

Retinitis Pegmentosa is a group of an unusual, genetically disorder that affects the

eye and causes rods and cones cells loss in the retina, due to the progressive atrophy of the
photoreceptors that limits the retina's ability to sense light. This disease causes a slow vision
loss, starting with decreasing night and sides' vision, resulting in an impairment vision [2].

Retinitis Pegmentosa is a group of multiple diseases that involves both eyes, rather
than a simple single disease. It affects the function of the light response cells [11]. Several
researches are done in order to find what causes such a disease, but up to this date, the
only known cause is that it is a genetic disorder.

Figure 2.3: Normal Retina (Left) vs. Retinitis Pegmentosa (Right) [2]

It is very common that people with RP will be diagnosed in their early age. Symptoms
may not appear all at the same time, but may differ according to each case. Not forget to
mention that the sequence of appearing is almost the same for all cases.

The beginning of appearance starts with a low night vision, which means that the
light sensing cells that are responsible of vision progressively decay, and seeing through dim

15
light become a real challenge to them. Later on, the patients start to lose lateral vision
(tunnel vision), eventually extending into the central visual field, as if they are looking
through a straw. It is worth to highlight that they do lose their peripheral vision gradually,
by the time they reach blindness [12]. In addition to that, the central vision reflects a blurry
scene. Visual sharpness -more likely to say shade vision- might get compromised because of
going with abnormalities in the cone photoreceptor cells, which would be the answer for
the difficulty to differentiate between color levels, and visual acuity. Presently, the eyes
become more sensitive to light. The prediction of the vision loss range or how does it
proceed is very difficult when the patient has retinitis Pigmentosa. Therefore the eye
specialist monitors the retinal cells health and does some tests to define how well the
patient sees.

It is important to realize that RP has no specific cure, provided that Vitamin A

therapy has been the best solution for the patients, it will help them living with RP for 10
extra years before reaching blindness [12].

With RP patients, it is hard for the doctor to determine the extent of vision loss or the
speed of such process. He will keep monitoring the eye condition and with some several
tests he can recognize how well the patient will be able to see.

2.3.2 Glaucoma

Specialists define Glaucoma as group of eye disorder that affects the optic nerves,
the one responsible for carrying information from the eye to the brain. Some time it’s called
the silent disease, since there are no initial symptoms appears at the very begging stages. It
is considered a huge vision destroyer, and statistics show that it is one of the strong reasons
that cause blindness in US [13]. It is detected through an eye exam that includes examining
the vision field and acuity [14].

Scientists refer this damage to the high-pressure level in the eye. In open-angle
glaucoma, the point in your eye where the iris attaches the cornea is as open as it has to be,
yet the eye's waste canals get to be stopped up after some time, bringing on an increment
in initial eye pressure and resulting harm to the optic nerve [14]. Glaucoma may reach you if
your parents have the sickness, on the off chance that you are African-American or Latino, if
you are diabetic or have cardiovascular illness. The danger of glaucoma additionally
increment with age [13].

16





Figure 2.4: The developing stage of the vision for a person with glaucoma [13]

Type 1: Usually the optic nerves are totally damaged when the patient start noticing
the symptoms. In the most spread type of glaucoma “Chronic Glaucoma”, peripheral vision
damages without any notice from the patient , and in most cases when they reach the
doctor, around 50% of their vision is almost gone [13]. Vision acuity “sharpness of vision” is
also maintained at the late stages of the disease.

Type 2: In Acute Glaucoma, sudden rises in eye pressure could cause a blurry vision,
pain, and colored holes around the light. Losing peripheral sight gradually and feeling blind
spots should be considered also as main symptoms [14].

Since vision loss of Glaucoma is irreversible; the objective of any treatment is to

anticipate loss of vision. Glaucoma can be treated with eye drops, pills, laser surgery,
conventional surgery or a mix of these systems. The uplifting news is that glaucoma can be
overseen if distinguished in the early stages and that with restorative and/or surgical
treatment; a great many people with glaucoma will be able to keep their sight [11].

2.3.3 Cataracts

Cataracts, the world's heading reason for blindness, it is difficult to identify the risk
factors of cataracts. It causes impair vision due to the clouding of the lens of both eyes [15].
People with cataracts have some difficulties in reading under low light; moreover their
shade recognition may be altered [1].

A cataract is an advanced clouding of the lens that causes no pain. It makes

difficulties to the patient to see clearly, it also block and prevent the light to be transited to
the lens. Therefore, all they can see is unclear, blurry pictures. Note that, Cataracts lead to
blindness by time [1].

As known that healthy lens is transparent; therefore lights can travel through the
lens easily. There are many reasons compile as we get older to cause cloudy areas in the
lens, that's why the lens can no longer transmit the light through the lens easily. Cataracts
have many reasons abbreviated as follows:

• Age related; Note that, the greatest risk factor of cataract is age
• Congenital

17
 • Genetic (family history)

Figure 2.5: blurry and indistict image results while having a cataract [1]

Researchers predicted that the number of people who will be affected by cataracts
would increase to 30.1 million in the next 20 years.

Symptoms and side effects normally creep up numerous years after onset, generally
when the person grows up. Progressively, a great amount of the lens becomes shady and
cloudy. Cataracts often influence both eyes, but infrequently the same [14]. People with
cataract may have the accompanying symptoms:
• Blurry vision.
• The patients see small patches blur part with their field of vision.
• Difficulty in reading under low light.
• Bright lights affect the vision.
• Unclear and faded colors.

The treatment of such cases differs according to the activities that the patient is
doing. For reading amplifying glasses must be worn, and to decrease the glare, sunglasses
will be helpful. Note that, these measures are temporary; cataract will keep developing and
progressively impair the vision more [15].

2.3.4 Diabetic Retinopathy

Diabetes is a condition where the measure of sugar in the blood is too high that the
body can't utilize it appropriately. Diabetic Retinopathy affects up to 80 percent of all
patients who have had diabetes for 10 years or more. More than 30 % of diabetic patients
have some type of diabetic eye ailment, for the most part diabetic retinopathy (DR). It is
also the main source of preventable visual impairment among people of working age, and a
noteworthy reason for sight loss for older people. It’s the damage in the retina due to
diabetes that can in the long run lead to blindness [16].

18
 Figure 2.6: A person with Diabetic Retinopathy will see blurry and distorted images (left) [16]

Diabetes occur when the body is unable to use and store sugar, elevating the sugar
level which damages the blood vessels in various organs like the heart, kidneys and eyes.
This can prompt loss of focal and perhaps peripheral vision.

In the first stage of the disease (non-proliferative Diabetic Retinopathy NPDR), there
will be no indications. Patients can have perfect visual acuity and the disease can only be
detected using funds photography that enables viewing the small blood-filled bulges in
artery walls. Fluorescein angiography can be utilized in case of low vision to see the back of
the eye showing any narrow or blocked blood vessels which indicates lack of blood flow
(retinal ischemia).

The second stage is called proliferative diabetic retinopathy (PDR), where abnormal
blood vessels form at the eye’s back burst and bleed causing blurred, fluctuating vision or
vision loss.

A standard treatment for DR is Laser photocoagulation; it delivers intense light to

motivate the blood vessels to shrink. Laser photocoagulation is performed in the workplace
setting with the patient situated before the laser unit. A few individuals may encounter
discomfort during the laser photocoagulation, however by and large it is a well-endured
office methodology [17].

2.3.5 Age-related macular degeneration

Age-related macular degeneration is a typical eye condition and a main source of

vision loss among individuals of age 50 and older. A degenerative disorder of the macula
(center of the retina), the eye's piece is required for sharp, focal vision, which gives us a
chance to see objects that are straight ahead. For some, AMD elevates rapidly and may lead
to sight loss while for some others it progresses slowly that the sight loss will not occur for a
longer period of time. The disease itself doesn’t lead to complete vision loss but it can
interfere with the daily activities like reading, driving, seeing faces or cooking [18].

Age is the most common risk factor for AMD and it’s most likely to happen after age
60. And some other factors include smoking, which doubles the chance of having AMD. Also,
studies show that race is also a factor where Caucasians are more likely to have the disease.

19
Genetics also are considered as a factor of getting AMD where the risk is higher when there
is family history. There are nearly 20 effective genes, which may increase the risk of having
AMD, and more are suspected.

In its initial stages, age-related macular degeneration might not have side effects and
may be unrecognized until it advances or influences both eyes. First indication of macular
degeneration is that lines are distorted. This may elevate to a continuous loss of focal vision.
Symptoms also include straight lines begin to seem twisted and dark, hazy regions or white
out shows up in the focal point of vision and rarely a change in colors [19].

Figure 2.7: AMD cause a blind spots and blurry image (Right) [19]

20
 Chapter 3

3 Related work
3.1 Introduction
Visually impaired people suffered for a long time, many ideas and techniques came
to the surface in order to simplify their lifestyles, but yet visually impaired and especially the
low vision people couldn’t manage the life in an easy way and independently as they
supposed to do.

Here is an overview of the field, several techniques and ideas that are presented
with an explanation of the advantages and disadvantages, and the reasons that make us
chose a totally different technique. This chapter discusses the previous works done to help
the visually impaired, and it is organized as follows: section 3.2 and 3.3 explains the
ultrasonic sensor and the infrared distance sensor respectively. Section 3.4 mentions some
assistance aids such as the smart cane and some applications with sensors. And finally,
section 3.5 lists some of the technical designs in the market such as the Google glass and the
orCam and depth map smart glass.

3.2 Ultrasonic sensor

First that comes to mind while thinking of a project that aims to create a reaction to
avoid obstacles is the ultrasonic sensor. It is almost everywhere with different ranges and
sizes and they differ by some minor details; the Seeedstudio sensor is a good example to be
examined.

Figure 3.1: Seeedstudio Ultrasonic [20]

21
 The Seeedstudio ultrasonic range finder (shown in figure 3.1) is a non-contact model
that measures the distance of the objects from a certain point using special but simple
criteria. It is compatible for electronic blocks, and manufactured to serve the project
modules for different industrial aspects.

The sensor works by sending out an ultrasound signals and receiving the echo after it
is reflected from the obstacles. Using the speed of sound and the time it takes to travel from
the objects, distances can be calculated accurately.

A paper published in October 22, 2011 mentioned that the sensor has a high-level
accuracy, long detection range and can identify the near obstacles easily, but one of the
disadvantages they faced was the wide beam of the sensor which makes it hard to
determine were the barriers are located exactly. The sensor may also detect what is called
“ghost echo” and it is defined as the reflected waves not by the wanted obstacle but by
unwanted objects such as walls, this may be misleading for the visually impaired person
[20].

3.3 Infrared distance sensor

In the infrared distance sensor, a small procedure happens to calculate the distance; a
bundle of light is reflected from the object onto the position-sensitive detector (PSD). The
distance that is detected depends on the location of the light, within the sensor this location
is transformed onto a voltage, and by using analog to digital converter for example; the
distance can be calculated mathematically.

These infrared sensors are inexpensive, consume almost no power, and can be
placed anywhere easily since it is very small, but there are many disadvantages for such
sensors. Many objects do not reflect light, so using this sensor in an application for the blind
people is risky since the sensor will not identify some of the obstacles, plus that the readings
that come out of two-colored surface at the same place may differ.

In addition, the range detected by the infrared sensor is very small, and for this blind
should be very close the obstacle to be able to avoid it while the goal of the project is to
make them be aware before reaching the objects [21].

3.4 Assistance aids

Through all the wide possibilities, visually impaired people have different cases, and
with each case the suitable assistance tool should be chosen perfectly to fit the user’s
needs.

22
3.4.1 Smart cane

Figure 3.2 : The Blindsport cane structure [22]

The first thing comes to mind while mentioning the visually impaired is the cane that
they carry all time in order to prevent collision with the obstacles. However, scientists and
engineers work hard in order to develop smart canes with various specifications to ensure a
safe walk for the user.

A project team advised by Donn Koh in University Of Singapore came out with a
white smart cane ”The Blindspot cane” (showed in figure 3.2) that can detect the objects
using sensors hanged to the cane and as they mentioned, this white cane can ensure a safe
path for the blind. The cane helps also to locate and identify the nearby friends, using the
GPS and Wi-Fi systems, a friend who is using their smart phone, can be easily recognized
and the user will have the ability to meet them. The feedback of the location of obstacles
and nearby friend is easily transferred to the user through an audio Bluetooth earpiece.

Type of cane used differs according to the sight loss, a person who is totally blind or
his vision is too low needs a long and strong cane in order to navigate through the new
environments by themselves with minimum collisions as possible. In the other hand, a
person with poor vision but not severe damages can use a lighter and shorter cane [22]. But
yet, why choosing to carry canes when we can get rid of such devices completely?

3.4.2 Applications with sensors

3.4.2.1 Glasses

The Idea of these glasses is to use the sensors again which are implemented in the
body to ensure getting the maximum amount of field information. If we take Theiavision
glasses as an example, the distance information will be presented to the user through a
sound generator that is connected to the microcontroller output pins [23].

There is also a Greek student who won the local award in Google’s 2014 Science Fair;
he presented a prototype of glasses with implemented sensors to detect the environment,

23
he wrote the code for the system by himself and transfers it to the glass. As an output of the
system, he added a vibration motor to the side of the glasses instead of sound commands,
to make it easier for the blind to interact with the environment without any sound
distortion [24].

This method may perfectly suit the totally blind people, but for this project cases,
people with low vision can identify the surrounding environment by a completely different,
better and easier criteria. In addition, adding a vibration motor can harm the brain since the
blind will keep having a feedback almost all the time.

3.4.2.2 Gloves

Similar to the glasses with the sensors and vibration methods, the idea of the gloves
also came to the surface. The wrist-mounted gloves helps the user to navigate safely in a
complex environment; it has ultrasonic sensors that detects the distance of the obstacle up
to 10 ft and converts the data into a pressure applied on the wrist, the closer the wearer
comes to the object, the faster the vibrations are [25].

Figure 3.3: Prototype of gloves with sensors [25]

3.5 Technical designs in the markets

Several companies took the responsibilities of helping the blind and low-vision
people, and many designs were developed in order to simplify the life for this category.

3.5.1 Google glass

Google glasses are a wearable and unique headset that looks like eyeglasses that can
communicate with the wearer using voice command. The glasses contains a small screen at
the corner of the frame to allow the user to stay in touch with the surrounding world, they

24
will be able to login personal emails, take pictures, communicate with friends, answering
calls, and reach the desired places using the integrated navigation system [26].

The company tried to help the blind people also, and many experiments were held in
order to check if the glasses could serve the low-vision people or not. University of Navada
has received the funding from Google glasses’ team to develop the glass in order to fit the
blind person needs and to help to reach their destinations without any help. The proposed
navigation system is indoor, after the user indicated were to go, the voice command will be
given in order to help the user through their route [27].

But still, the developed idea requires a cane, and as we mentioned previously, we
want to get rid of any additional tools in order to make the low-vision person looks like a
person with a normal vision.

Figure 3.4: Google glass [27]

3.5.2 OrCam

Figure 3.5: OrCam [28]

The establishment of the orCam was in the 2010, the significant value of the orCam
is to help and provide independency to the visually impaired and the blinds. It is a high
technology system with many different features to facilitate everyday tasks and to help

25
them exploring the surroundings. OrCam assists the visually impaired and blinds to
understand text and read it to the wearer using a small size earpiece that does not ban the
hearing, distinguish and differentiate faces that works on memorizing. A bright small camera
is mounted on the right frame of the glass is programmed in a way to do all the features.
OrCam doesn’t need an internet to move freely outdoors, it uses the internal memory that
has the ability to remember and recall hundreds of saved matters [28].

3.5.3 Depth based smart glasses

Unlike the previously mentioned ideas and techniques, this head mounted visual
display uses the user's remaining vision. Prof Stephen Hicks and his team from the
University of Oxford came up with an idea of a head-mounted glass that uses the remaining
vision, where this smart glass has a depth camera attached to a transparent display and an
earpiece all combined to provide the user a convenient movement avoiding any obstacles.
The transparent displays were custom-made so the user will see the depth map image from
the inside and others will see the user wearing a transparent glass. The depth camera helps
the user to see nearby objects in a higher light intensity while the far objects will be unseen
clearly.

The down-side of this visual aid cannot help the user outdoors where sunlight exists,
that’s because the used depth map camera cannot give an output in a bright environment,
and moreover, the user must carry a laptop all the time in order for the camera to process
the surrounding scene and generate a corresponding depth map.

As seen from the results in figure 3.6, time spent to get familiar with the glass and walk
comfortably in an unknown environment decreases logarithmically, user's velocity increases
linearly and number of obstacles collisions decreases linearly [29].

The idea of the proposed design -NOOR- resembles Prof. Hicks works with
differences, and mainly NOOR works indoors and outdoors using ZED camera. Furthermore,
instead of allowing a user to carry a laptop while moving, Jetson TK1 is used which is a
12x12 cm CPU that can be easily programmed. Not to forget to mention that the tests
criteria in this report will follow Prof. Hick’s tests for comparison reasons.

26


Figure 3.6 : A) Prof. Stephen Hicks mart glasses with a laptop that should be carried all the time. B)
Different mazes of a single corridor for tests purposes. C) Time, velocity and total collisions vs.
number of trials graphs. [29]

27
 Chapter 4

4 Synthesis
4.1 Introduction
The main idea of the adopted solution is to extend the vision sense through an
interface. The development of a glass system to those who retains low-vision needs
basically three main elements: CPU, depth camera and a display, which should be chosen
accurately to insure that the users get the best quality they can get.

This chapter is organized as follows: section 4.2 mentions the different alternatives
of the depth cameras with their specifications. Section 4.3 mentions the differences
between the two kinds of displays and shows their advantages over each other and presents
the selected one. Section 4.4 concludes the chapter by mentioning the chosen depth
camera and display.

4.2 Depth cameras

Recently, Orbbec, Xtion, Kinect and ZED show widely at the amusement market.
Additionally, the results and outcomes assure that these cameras can be utilized within a lot
of engineering applications when the range is short and precision necessities are not
extremely strict.

3D physical objects are digitalized using an engineering appliance which is a laser

scanner. Those 3D laser scanners use various sensor technologies relying on the matters of
the measurements. Laser scanners are utilized for small ranges and high accuracy and
precision. The scanner radiates a laser on the exterior of the objects, and exploits a camera
to search for the laser dot's location. Relying upon how far the laser hits the exterior, the
camera sensor shows different places of the laser dots.

Orbbec, Xtion, Kinect and ZED cameras are being significantly important 3D cameras.
These cameras merge two techniques of the classic computer vision with the structure light,
which are: the depth from focus and stereo. The system uses an infrared laser with a dotted
manner or uses the stereo-vision principle. Moreover, Orbbec, Xtion, Kinect and ZED are of
the most essential 3D measuring devices for objectives recognitions, spectacle
reconstructions, mapping, and robotics. The utilization of Orbbec, Xtion, Kinect and ZED
cameras in some applications other than entertainment is of an extremely great interest,
nevertheless, to ensure that its application in other areas where the objects geometric
properties are important, the technical properties of the system must be studied [29].

28
 This section draws an analogy of the four cameras and compares between the
optimum results of different developers and the gotten results from the tests done on the
chosen camera.

4.2.1 Orbbec
According to the design, the depth camera used in the smart glass should give a high
quality grey-scale output, with a high frame rate per second while maintaining a small
weight and dimensions, and reflect excellent images of the surrounding environment. Such
requirements are available in several depth cameras; one of them is Orbbec.

The company developed until now three different designs, with different
specifications, this allows the designer to choose their own type to fit the project. The
outside shape differs between them, the sizes, and some minor and interior specifications,
but they almost follow the same method in analyzing and examining the field, and they give
approximately the same output [30].

4.2.1.1 Orbbec persee

Persee, as shown in figure 4.1, is a 3D camera that is additionally a completely

working PC with a powerful ARM processor and built-in Astra Pro 3D camera. It is a creative
combination that doesn’t need to waste time getting set up [30].

The specifications of the orbbec persee make it a very special device that can be
used in many applications rather than the entertainment applications. Table 4.1 shows the
different specifications of it.

Figure 4.1: Orbbic Persee [30]

29
 Table 4.1: The specifications of Orbbec Persee

Size and Dimensions 18.4 x 3.5 x 4.6 cm

Weight 800 g
Range 0.4 to 8 m
Size of depth image 640x480 VGA – and 16bit at 30fps
Resolution 1280x720 at 30fps
Field of View 60° H x 49.5°V x 73°D
Software Orbbec Astra SDK and OpenNI
Power Adapter 5V – 3A
Working Area Indoor

4.2.1.2 Orbbec Asro Pro

In addition to the built-in camera-computer that is available in Persee, Astra Pro, as

shown in figure 4.2, is recognized as an excellent standalone 3D camera which consists of
the Orbbec chip and a 720p color sensors. Due to the records, it has an incredible accuracy
level, a good quality of depth resolution, with a 0.4m - 8m range. Orbbec Astro Pro has
many interesting specifications listed in table 4,2.

Table 4.2: Orbbec Astro Pro Specifications

Size and Dimensions 16.0 x 3.0 x 4.0 c

Weight 600 g
Range 0.4 to 8 m
Size of depth image 640 x 480 VGA - 16bit at 30fps
Resolution 1280x720 at 30FPS
Field of View 60° H x 49.5°V x 73°D
Software Orbbec Astra SDK and OpenNI
Working Area Indoor

Figure 4.2: Orbbec Astra Pro [30]

30
 4.2.1.3 Output form

What makes the Orbbec 3D technology a good choice is the output quality. The
depth camera’s output is represented in figure 4.3. Brightness, color levels, and the amount
of the represented details, all will be adjustable so it can fit the cases and help them in a
very efficient and specific way.

Figure 4.3: A human standing at 6m distance can be clearly distinguished in the
Depth image (left) and color image (right) from the wall behind her at 8m distance [31]

4.2.2 Microsoft Kinect

The Microsoft Kinect camera combines several developed sensing hardware.,

especially, RGB color camera, depth sensor, and four-microphone array that show a capture
of a 3D motion of full-body, objects recognition, and a capability of voice recognition (Figure
4.4). The depth sensor comprises of an infrared projector joined with an infrared camera,
which is a monochrome CMOS sensor.

Infrared RGB Infrared

projector camera camera

Figure 4.4: Microsoft Kinect sensor with the cover taken off [32]

Figure 4.5 illustrates the depth map generated by the kinect sensor. The value of the
depth map is encoded only with grey-scale; dark pixel indicates that the point is located
closer to the camera. Where the black pixel means that there is no depth value for it, this
case could happen if the points are located too far where the values of the depth map are

31
not accurate, are located too close where the field of view of the infrared projector and
infrared camera is limited and it enters the blind region [32].

Figure 4.5: Microsoft Kinect sensor depth image [33]

The outcome of a kinect sensor is a 2D video and 30fps depth images. Although, the
depth information of the kinect sensor is not extremely accurate. Figure 4.7a, illustrates an
example of a captured information by kinect sensor. Figure 4.7c, shows a close-up of the
face, where the information is noisy [34].

(a) (b) (c)

Figure 4.6: Human face captured with Microsoft Kinect sensor. (a) Video frame, (b) depth image, and
(c) a close-up of the facial surface [34]

As any other cameras, Microsoft kinect has many specifications to be taken into
consideration, table 4.3 shows the different specifications.

Table 4.3: Microsoft kinect specifications.
Kinect Array specification
Angle of viewing 43ᶱV , 57ᶱH
Dimensions 15.0 x 6.1 x 4.8 cm
Default resolution VGA (640 x 480)
depth stream
Working Area Indoor

32


4.2.3 Asus Xtion Pro

Xtion PRO LIVE uses infrared sensors, adaptive depth recognition, colored picture
detecting and sound stream to catch a client's ongoing movement and voice, giving an exact
tracking. Xtion also has a development solution that allows developers to have many
applications related to the moving gestures using motion-sensing technology [35].

Figure 4.7: Asus Xtion [35]

Xtion is compatible with OPNI NITE middleware SDK which makes the motion-
sensing applications and games development much easier, which is beneficial to have an
easier programming language. One of the most important features its high distance range
that it can reach where the range starts from 0.8m up to 3.5m.

Xtion tracks people's movements immediately; it has more than 8 predefined

postures to permit users to push, click, circle, wave and much more which makes it perfect
for controlling a user interface. It also permits designers to track a clients' entire body
motion. Xtion PRO LIVE empowers color (RGB) picture detecting. With RGB, the product can
catch the clients' picture, which is valuable for human recognition. The Xtion PRO LIVE
features an easy plug and play USB design, which is compatible with the OPNI NITE
middleware, making it very easy to use [35].

Table 4.4 shows the main different specifications of Xtion Pro Live Asus to be
compared with the other camera's specifications [36].

Table 4.4: Asus Xtion specifications

Xtion Array specification
Distance of Use 0.8 to 3.5 m

Field of View 58° H, 45° V, 70° D

Depth Image Size VGA (640x480) at 30 fps

Resolution SXGA (1280*1024)

Software software development kits (OPEN NI SDK bundled)

33
 Dimensions 18 x 3.5 x 5 cm

Weight 540g
Working Area Indoor

4.2.4 ZED

ZED is a stereo camera with a small-weight depth sensor that uses the passive
stereovision technique, and the output of the system is a video with high resolution and
presented using USB 3.0.

ZED SDK is the software that helps to construct the depth scene from the side-by-
side video cameras; the graphic processing unit in the machine also helps to do that [37].

Figure 4.8: ZED stereovision camera

4.2.4.1 Specifications

ZED camera can work in both environments, indoor and outdoor. Other than that,
regarding the output resolution, each quality requires a certain video mode and gives
different value of fps, table 4.5 explains this more clearly.

Table 4.5: The output resolution of ZED
Resolution Frame Per Second Video Mode
4416x1242 15 2.2K
3840X1080 30 1080p
2560x720 60 720p
1280x480 120 VGA

The depth resolution is the same as the selected video resolution, the range starts
from 1.5m up to 20m, and the depth format is 32-bits with OpenCV compatible which
allows detecting and recognizing faces, motions, constructing 3D modules, and adding the
images together so they can build a high resolution scene. The lens of the cameras is a wide-
angle, made of glass and provides 110 degree as a maximum field of view with aperture
f/2.0.

34
 In order to fit all the conditions and to improve the performance of the camera in
the low-light areas, people who work on ZED added a backlight illumination sensor, which is
a digital image sensor that uses a special arrangement to increase the light that enters the
camera, this technique is mostly important in the low-light environments. The sensors of
ZED are 1/27” and have been chosen perfectly with a resolution 4M per sensor [33].

Controlling the camera is one of the main specifications that any developer is looking
for. ZED team made it very easy to adjust so it fits the user needs. The resolution,
brightness, frame rate, saturation, contrast, sharpness, and white balance, all can be
controlled somehow without adding any external item. Table 4.6 describes the physical
specifications of ZED camera

Table 4.6: Physical specifications of ZED

Size (LxWxH) Weight Power consumption Operating

Temperature
17.5x3.0x3.3 0.159 kg 380mA/5V 0 – 45 degree C

4.2.4.2 Output form

The 3D depth map of ZED corresponds to the distance from the stereo camera, to
put it differently, near objects are reflected with the same shape but in white, each time the
object gets further away from the camera the white reflection decreases and the object
enters the grey-scale region. The black background of the picture represents objects that
are at a distance more than 20m that cannot be detected by the camera.

Figure 4.9: original image and the depth map corresponding to it using ZED SDK with OpenCV library

Writing a code using the ZED SDK software, where the original image and its output
are shown in figure 4.9, can produce the grey-scale depth map output.

35
 4.2.4.3 How it works

Figure 4.10:How a ZED camera works

Figure 4.10 shows how a ZED camera works. Firstly, the ZED records the scenes using
the left and right sources of cameras in a very accurate way and sends the video scenes
through a USB 3.0 interface. Secondly, the software (ZED SDK) has an internal graphic chip
that comprehends the texture of the delivered video scene and calculates the depth map of
it in the real time. Finally, the output of the ZED SDK can be implemented for different
purposes and applications [37].

4.2.4.4 Block diagram

3D generation process

Left
Stereo source Image Image 3D map Depth
analysis Correction generation
map
camera
Right
source

Figure 4.11: A block diagram of using ZED to generate a depth map

Figure 4.11 represents the steps of generating depth map in the camera. ZED has
two cameras (left and right sources), each source captures the same scene in a different
angle to produce a full scene when collaborating, the scenes enter the 3D generation
processes, where the image analysis block detects the roll, tilt, height, focus and depth, it
sends the signal to the image correction block, where it matches the colors, and sends the
signal again to the 3D map generation where it generates a dense and a semi-dense maps.
The result of the three blocks is a 3 dimensions depth map.

36
 4.2.4.5 Experimental results

Due to the different features of the ZED stereolabs camera, numerous characteristics
can be taken into considerations. As it's known that the ZED works with ZED SDK together
with a third party. The project focuses on two main libraries, which are: OpenCV and
OpenGL provided that all codes are written in C++ language.

4.2.5 Alternatives comparison

Table 4.7: Alternatives main differences

Dimensions Range Working area Frame rate Interface
Orbbec 18.4 x 3.5 x 4.6 0.4 – 8m Indoor – day 30 fps USB 2.0
Kinect 15.0 x 6.1 x 4.8 0.8 – 4m Indoor – day 30 fps USB 3.0
Xtion 18.0 x 3.5 x 3.0 0.8 – 3.5m Indoor – day 30 fps USB 2.0
ZED 17.5 x 3.0 x 3.3 1.5 – 20m Indoor - outdoor >= 30fps USB 3.0


The main goal of this section is to compare between four strong 3-D depth cameras
in the market; Orbbec, Xtion, Kinect and ZED. After investigating about each and every
detail of the four types, the best alternative that suits the project is ZED since it meets the
needed specifications with the largest range from 150cm up to 20m, however, while testing,
the result shows that the camera can detect up to 15 m. It also gives high quality in
transferring information. Moreover, it works in indoor and outdoor environments, while the
other three cameras work only in the indoor environment. Additionally, sense the ZED has a
back-illumination sensor; it can work at night where it increases the number of light caught.
Therefore, the idea of using the camera for the indoor use and ultrasonic sensors for
outdoor use can be eliminated.

4.3 Jetson TK1

While testing the project it had been noted that the processor of the camera could
not process the images. So, the best choice was using an external CPU, which contains a
strong GPU. Obligate the user to carry out a computer device is unpractical. That’s why, with
the variety of options in the market, the NVIDIA Jetson TK-1 was the best and most suitable
choice for many reasons and this section will explain the features it offers, the specifications
and how it is used in the project.

Jetson is a visual computing platform featuring a Tegra K1 SOC (CPU+GPU+ISP in a

single chip) and comes usually with a Linux environment. Supporting many API’s
(Application Program interface) and it gives a variety of hardware interfaces making the
jetson an extensible, flexible tool. It is an ideal low-powered device for high performance.
Jetson TK-1 will be used to operate the ZED camera and a rechargeable battery will be used

37
in order to make the design more efficient and user friendly. Figure 4.12 shows the Jetson
TK1 and table 4.8 shows the specifications of the Jetson TK-1.

Figure 4.12: Jetson TK1

Table 4.8: Specifications of Jetson TK1
Dimensions 5" x 5" (127mm x 127mm) board
GPU NVIDIA Kepler"GK20a" GPU with 192 SM3.2 CUDA cores (up to
326 GFLOPS)
CPU NVIDIA "4-Plus-1" 2.32GHz ARM quad-core Cortex-A15 CPU with
Cortex-A15 battery-saving shadow-core
Storage 16GB fast eMMC 4.51 (routed to SDMMC4)
USB 3.0 A full-size Type-A female socket
HDMI A full-size port
Power A 12V DC barrel power jack
Camera ports 2 fast CSI-2 MIPI camera ports
LCD port LVDS and eDP Display Panel
Hardware- • CUDA 6.0 (SM3.2, roughly the same as desktop SM3.5)
accelerated APIs • OpenGL 4.4
supported: • OpenMAX IL multimedia codec including H.264, VC-1 and
VP8 through Gstreamer
• NPP (CUDA optimized NVIDIA Performance Primitives)
• OpenCV4Tegra

38
 In order to get the Jetson and ZED working together there were some steps for
setting up. First of all Linux operating system must be installed, followed by the Jetson
development pack. Then, enabling USB 3.0 since it is recognized by default as USB 2.0. And
the last step is installing the ZED SDK; this enables the user to use the ZED camera using the
Jetson without the need of large computer devices. And finally, showing the depth map
using different libraries.

4.4 Displays
To get the best fitting display on the glass, looking through variety of choices was an
enormous challenge, each had advantages over others and disadvantages too. In this
section the best alternatives are highlighted and the focus is on the main differences
between them in order to be able to choose the right display that fits the specified
requirements.

4.4.1 OLED

The organic light emitting diode (OLED) is made by two separate conductors, which
are filled by progression thin films that are organic. Once an electrical current passes
through it, the OLED emits bright light. Backlights are not required for the OLED since that it
emits its own light. OLED has many advantages over LCD. It consumes a lower amount of
power; battery contrast is faster, as well as the refresh rate, and a considerably greater and
better brightness [38].

A flexible 1.66" OLED display is one of the options considered for the project; it is a
parallel interface small screen, perfect for different applications, such as: smart phones,
modern smart watches or even for wearable watches. It can easily bend to fit different kinds
of projects. Figure 4.13 shows the flexible OLED [39]. Table 4.9 shows the different
specifications of each item mentioned [39].

Figure 4.13L Flexible OLED [40]

39
 Table 4.9: 1.66" flexible OLED specifications
Item Specifications
Dimensions 32.2 x 54.99 x 1.18 mm (W x H xT)
Dots Number (Resolution) 132 x 176 x RGB ( W x H x RGB)
Viewing Area 26.314 x 34.762 mm (W x H)
Weight 0.8 grams
Current Consumption – Logic 1.5 mA (Typical)
Current Consumption – OLED 37 mA (Typical)

4.4.2 TFT

To improve the quality of the image and the contrast, a thin-film-transistor LCD (TFT
LCD) is being used widely as an efficient display. It is an active-type matrix that is
implemented in a huge range of applications such as mobile monitors, TVs, GPS monitors,
digital cameras, and a lot of video games as well. Not to forget to mention that they are
available with different types, each has its own functioning system.

What makes the TFT a perfect choice for any consumer is the variety of the electrical
interfaces that can be connected to. It accepts HDMI, VGA, RGB, DVI, or DPI, each according
to the chosen TFT. To convert these types to a digital RGB while maintaining the original
resolution of the display, manufacturer implemented a converter in a control board, which
is connected to the interface [41].

Choice 1:

As a strong option, a 2” TFT LCD model with an RGB interface will perfectly fit the
design (Model no. ZP20053), it has a good resolution. Table 4.10 describes the technical
specifications and figure 4.14 shows the TFT LCD model with the dimensions [42].

Table 4.10: TFT LCD model technical specifications [42]
Resolution (RGB) 480×240
Size (cm) 4.61 × 4.096 × 0.253
Active area size (cm) 4.056 x 3.048
Interface RGB ( 8-bits )
Color Depth 16.7M
Number of pins 40

40

Figure 4.14: TFT LCD model

Choice 2:

Adafruit 5” is a display (figure 4.15) with smaller than usual HDMI screen and an
implicit touchscreen. Very simple to use, and the user can utilize such display with any PC
that has HDMI interface. The 5” display has a sufficient resolution (800x480), which makes it
very useful to run most of the programs and software, and with this efficiency it can be
implemented in a wide range of embedded projects. One of the advantages is that this
display can be run using a USB cable as long as the used CPU can support 500 mA.

Figure 4.15: Adafruit 5" display

There are two versions of such screens, touch and non-touchable screens. The
display has good specifications that match the proposed design and table 4.11 describes
them [1].

41


Table 4.11: Adafruit display specifications
Size (cm) Weight Type Power Operating
Consumption Temperature (°C)
12.6x7.6x0.7 0.106 TFT 500 mA 0 – 70

The Display has a backlight that helps to increase the brightness and makes the
screen looks better. Enabling the backlight will consume more power and thus will reduce
the battery lifetime [1].

4.5 Battery
A Li-Po battery is a shortcut of Lithium Polymer battery. The one used in this project is
a 2200mAh rechargeable battery with 3 cells, each cell needs 3.3V and the total amount of
voltage of all three cells is 11.1V, once charging, it should not exceed 12.6V, and once
discharging, it should not be less than 9.9V. Fully charging the battery from 9.9V to 12.6V
needs around an hour and works for 6 hours, if the same battery is connected to Jetson TK1
and the display it takes more power therefore it discharges faster, where it takes around 2
hours to be fully discharged.

4.6 Conclusion
After a long study and comparing many alternatives a conclusion can be drawn, the
depth camera which will be used in the design is the ZED because of its specifications that
meet the desired requirements, the Jetson TK1 will be added to the system to play the role
of the CPU, and the display that will play the role of the lens will be the 5” Adafruit display
due to the availability of the screen in the market, where it is going to replace the two small
displays in the previous proposed design.

42
 Chapter 5

5 Proposed design
5.1 Proposed solution

Among all the mentioned solutions, we adopted the idea of smart glasses, where the
user will move independently without any additional assistants such as canes or guide dogs.
The goal of the project is to help the low-vision user to avoid the obstacles surrounding
them in an unknown environment, and to be able to interact with different environments by
improving the user’s mobility. The main idea of the proposed solution is to extend the vision
sense through an interface.

The development of a glass system to those who retain low-vision needs basically
three main elements, which have been chosen accurately to insure that the users get the
best quality they can get.

In the first place, after analyzing the available sources in the market, differentiating
between varieties of choices was a big challenge. After all, the best camera that matches the
required specifications is ZED; it is carefully studied in the depth cameras section. The
general principle of the camera is to deliver an accurate perception in grey-scale 3D
mapping in real time using stereovision technique. Locating two stereo cameras that scan a
wide field of the environment with accurate distance measurements and send it to an
internal processor that cooperates with the Jetson TK1 processor to transfer the scenes into
a depth map format.

Not only the cameras and the Jetson TK1, but also a display is essential to play the
role of the glasses’ lenses. After investigating the specifications of many displays, the chosen
display is a TFT display that blocks the outcome light with the help of the glass’s body and
perfectly fits the shape of the structure design.

Equally important, the used imaging techniques will play the major role in the
design. In order to simplify the user’s life, few modifications must be done to the original
scene. By writing a certain programming code, the brightness, contrast, saturation,
sharpness, and white balance adjustments can be controlled, which will result in a new and
better image that will be reconstructed to suit the patient’s condition.

The design will allow the low-vision persons to experience the world around them
for the first time, where it aims to help the low-vision people navigate independently
through different environments while minimizing the number of collisions.

43
5.2 Hardware design
The glass’s hardware consists of two main parts and some subsidiary parts: the ZED
camera, a single TFT display, a box that contains Jetson TK1 and the system batteries, a
sponge supporting pads and the plastic body of the camera.

The zed camera plays a huge part in the system design where it is the main receiver
for the surrounding scene, wire connection connects the output of the zed camera to Jetson
TK1, which is located in the box where the battery and all necessary wires are also placed. A
hallow sponge is placed between the users eyes and the displays to provide maximum level
of comfort to the user, the sponge will block the user from seeing any outer scenes so there
will be no interference between what the user sees directly and what he/she sees in the
displays.

Figure 5.1: Hardware structure of NOOR

44

Jetson TK1

Left
rd
source ZED SDK + 3 party library

Depth
Image Image 3D map
Detection map
Camera (Video
analysis Correction generation
stream) Right
source

Display

Figure 5.2: The block diagram of the entire system design

The detailed block diagram of the system is shown in figure 5.2. Firstly, the camera’s
function is to detect different scenes and to help to process them in the same time; the left
and right stereo cameras do detecting, and the ZED SDK with the help of a third party library
such as OpenCV and OpenGL do processing. The main processing operations are handled in
Jetson TK1 since it has a very powerful GPU. In the processing stage, an image enhancement
code is written to the integrated GPU to change the properties of image and to calculate the
depth map by using the triangulation method. The end of this point transforms the image
into a depth map image. Later on, the image will be conveyed to the display using an HDMI
cable.

A box is added to system as mentioned to hide the Jetson TK1 and the battery. The
content of the box functions as the heart of the system; where it contains the most
significant components that allows the system to operate. Firstly, Jetson TK1 is placed in the
box to maximize the protection level, and according to the fact that it is unpractical to let
the users carry the Jetson TK1 while they are moving through an environment, therefore,
hiding it in the box is more convenient. Secondly, the Li-Po battery is also placed in the box
but is separated from the Jetson TK1 except a small hole that connects the charging wire
between them.

5.3 Software design

In order for the glass to function properly, a certain program for the ZED camera is
written where it is considered the base of the smart glass. This program will capture the
scene for the user and transform it from a normal scene to a 3D depth map; this procedure
will be computed in the graphic processing unit of the camera. The used programming

45
language is C++ language, and the code is written using a special program called ZED SDK
with a third party library.

5.3.1 ZED SDK

The ZED camera can be programmed using special associated software called “ZED
SDK”. It is a special program developed by the stereo-labs team to make it simple to serve
any application, to take the full advantage of the program a strong NVIDIA GPU is needed.
The ZED SDK works with some libraries from different programs such as OpenCV and
OpenGL. The program will process the different captured scenes by transforming the scenes
into a depth map that suits the requirements.

ZED SDK is able to compute three types of data:
• Side-by-side 3D RGB video stream on sides left and right.
• Depth map computed on the graphics-processing unit (GPU) and taken from the
video stream
• Confidence map that represents the accurate measurements

To be able to use the ZED SDK, the user’s operating system must match specific
requirements, with the following configuration, note that the main reason of choosing the
Jetson TK1 is that all the specifications are included in it:

• Dual-core 2,3GHz
• 4GB of RAM
• NVIDIA GPU with compute capabilities >2.0
• CUDA 6.0
• USB port 3.0
• Windows 7, 8, 8.1(64 bits) Ubuntu 14.04, L4T21.3/4

For having a high-resolution footage while recording the video, user needs 250 MB/s
at least as a transfer speed [37].

5.3.1.1 Experimental results

Due to the different features of the ZED stereolabs camera, numerous characteristics
can be taken into considerations. As it's known that the ZED works with ZED SDK together
with a third party. The project will focus on two main libraries, which are: OpenCV and
OpenGL provided that all codes are written in C++ language.

Three codes were written, each one of them gives different quality of output. Due to
the differences in the users medical situations, and after testing the glasses on the chosen
cases, the results proved that there is no fixed code for the glasses and any code of these
three may be the chosen one, depending on the user itself.

46
5.3.1.1.1 ZED with OpenCV

ZED with OpenCV demonstrates how to link the SDK with the library of the OpenCV.
A written code using OpenCV library can either conceive the depth or the disparity map.
Each camera of the ZED captures the same scene but in a slightly different angle, and the
default situation is a normal colored image.

Two codes has been written to represent the depth map with OpenCV library, each
one of them handles different situations, and figure 5.3 illustrates the flow chart of both
codes.

Setting ZED depth in Error

performance mode No (Quit)
No Output

Yes

No
Detect image size

Yes

Choosing wanted
resolution

Yes

No
Get right and left
image buffers

Yes

Measure disparity
map and retrieve
depth map

Figure 5.3: OpenCV codes flowchart

47
5.3.1.1.1.1 Code 1
A simple switch statement in the code can show different views using keyboard
shortcuts. The results and the shortcuts are shown in table 5.1.

Table 5.1: Key shortcuts to control different characteristics of depth image
Main Hotkeys Display Hotkeys
b: Decrease reliability index by 1 0: Left View

n: Increase reliability index by 1 1: Right View

r: Sensing Mode: RAW 2: Anaglyph View

f: Sensing Mode: FULL 3: Gray Scale Difference View

s: Display Confidence Map 4: Side by Side View

w: Save Side by Side image in PNG 5: Overlay View

v: Save Disparity image in PNG d: Toggle Disparity/Depth View

Figure 5.4: A combination of the left and right images, default in OpenCV

Figure 5.5: Left view (on the left), and right view (on the right)

Figure 5.5 shows that the left and right cameras capture the same scene but with
different angles as the human eyes work. The overlay image shown in figure 5.4 combines
both the right and the left images.

48


Figure 5.6: Overlay image of left and right sources

4.5 m


2 m 3.1 m

1 m





Figure 5.7: Depth map

Pressing on 'd' key toggles between the two maps, depth and disparity. Disparity is a
2D image that refers to the separation between two relating points in the left and right
pictures of a stereo pair, while depth is a 3D image contains the depth values for different
pixels.

Correspondingly, sensing modes can vary between FULL and RAW to grab the image
as well as the depth. RAW sensing mode is adequate for collision avoidance and navigation
applications, while FULL mode is targeted at augmented reality and computational imaging
where a fully dense depth map is needed.








Figure 5.8: FULL sensing mode (right), RAW sensing mode (left)

49
 Similarly, ZED can be initialized in a computation mode: performance, quality or
neither. Performance mode is the fastest, it needs less GPU memory, and the resulted
disparity map is less detailed, while the quality mode gives a better quality with a more
detailed disparity map.

Figure 5.9: Performance mode (left), Quality mode (right)

If an application needs to view the left and the right disparities jointly, pressing on ‘3’
can do it.

Figure 5.10: Left and right views in disparity

Decreasing and increasing reliability indices by 1 can also be done in both RAW and
FULL mode.

Figure 5.11: Decrease (left) and increase (right) reliability index by 1 in RAW mode

50









Figure 5.12:Decrease (left) and increase (right) reliability index by 1 in FULL mode

Using ZED with OpenCV gives good results with high resolutions and image quality
except that while video casting in real time, a noticed delay was detected in version 0.9.2,
the duration of the delay was around 1.5 seconds, which is much higher than the expected
delay and can’t serve the project purposes perfectly, but in version 0.9.3 the delay has
reduced by almost 70% where the delay became unnoticed.

5.3.1.1.1.2 Code 2

In this approach delay is around 1 second compared with code 1,the frame rates is
much better and gives continuous video stream, but the quality of the depth map is poor.

Figure 5.13: depth map represented by OpenCV.

5.3.1.1.2 ZED with OpenGL

ZED with OpenGL shows the most effective approach to get and show pictures and
disparity map with the ZED SDK. Since that there is no GPU-to-CPU read-back, it is expected
to be the best approach for real-time observing. The performance of the result in this
approach is much higher compared with the OpenCV sample. The delay in this case cannot
be noticed and the frame rates is higher, but with these results, the quality of the depth
map is reduced and figure 5.14 and 5.15 illustrates the outcome of the OpenGL code and
the flowchart.

51

Figure 5.14: depth map represented by OpenGL

Declare GL and ZED No

resources ERROR

Yes

Configure window
size
Yes

Set GPU buffer

Yes

No
Draw image texture
in left and right part

Yes

Measure disparity
map and retrieve
depth map

Figure 5.15: OpenGL code flowchart

52
5.4 Testing methods

Briefly, the system is a glass that bocks the outer environment and shows the scene
in a grey-scale depth map on a 5 inch screen that roles as the lens of the glass to help the
low-vision people. Tests were made on two categories: healthy people with normal vision,
and people with low-vision. The purpose of testing on normal people is to insure that the
system works perfectly and they can avoid the different obstacles while moving through an
environment, and on low-vision people to insure that the system can help them walking
independently with no assistants in a unique way. All participants: normal and low vision
people were asked to get used to the glass before using it, this is done by giving them a
minute or two to scan the environment before walking. The test environment was a narrow
corridor with different obstacles (chairs, vases, small tables and others), each participant
has walked in the same corridor for number of times with different obstacles arrangements.

5.4.1 Study 1: Obstacle avoidance test on healthy people with normal vision

Tests were done on five healthy persons with normal vision. The test had two
intrinsic purposes: foremost to define if the users are able to avoid the different obstacles
around them in a narrow maze, comparatively to measure the rate at which the users could
figure out how to utilize the unfamiliar screen. Users where directed on how to use the
system by allowing them to relax and scan the environment before moving around, else,
there was no other training or preparing for the test. Participants were requested to move
through the prepared maze that is a narrow corridor (approximately 9x2 meters) with
different obstacles without crashing or colliding the obstacle edges or hitting walls.
Rearranging the places of the obstacles created ten different mazes, and the users finished
each maze in a random route. Number of trials versus time to complete a maze, number of
collisions, and the user velocity were recorded.
All participants were able to complete all the ten mazes; figure 5.17 shows the
relationship between the number of trials versus time, collisions and velocity.

53


Figure 5.16: A photo of one arrangement of the different obstacles and schematics show the setup
up of the five mazes

54
 Trial Vs. number of collisions
2

Collisions 1.5
y = -0.0824x + 1.0533
1 R² = 0.30461

0.5

0
0 2 4 6 8 10 12
Trial

Trial Vs. Time

50
40 y = -1.9195x + 33.537
Time (sec)

30 R² = 0.748

20
10
0
0 2 4 6 8 10 12
Trial

Trial Vs. Velocity

40
Velocity (m/min)

30

20
y = 1.9145x + 14.504
10 R² = 0.87186

0
0 2 4 6 8 10 12
Trial

Figure 5.17: a) Number of trials vs. Collisions graph. b) Number of trials vs. Time graph. c) Number of
trials vs. Velocity graph, which belongs to the average results .

55
 The average time took both participants to complete maze 1 is 39.3 seconds while
only an average of 14.7 seconds took them to complete maze 10, as seen from the trials
versus time graph, the time is decreasing logarithmically by an average of R²=0.748.
Similarly, number of collisions in the graph decreased from an average of 1.6 collisions in
the first trial to 0.2 collisions is the last trial; at the same time the velocity of the participants
has increased almost linearly from an average of 13.74 m/min to 36.73 m/min.

5.4.2 Study 2: Obstacle avoidance test on low-vision people

With the co-operation of Emirates Association of the Visually Impaired (EAVI), Noor
has been tested on three healthy participants with different visual cases. Two of them were
diagnosed with Retinitis Pegmentosa, and one with Glaucoma, note that the two patients
with RP have different levels of vision. Table 5.2 illustrates the details of the participant
cases.

Table 5.2: Low-vision participants details

Participant No. Age Gender Medical case
1 18 Male Glaucoma
2 22 Female RP
3 25 Female RP

Before starting the tests on the participants, they were asked to get familiar with
the glass by scanning the surrounding environment, and few instructions were given to
them in order to teach them how to use NOOR. Participant 2 felt comfortable while using
the glass, she was able to sense the environment clearly and avoid different obstacles after
several trials, although participant 3 is diagnosed with the same disability but the glass could
not improve her movement rate, since she felt more comfortable while moving through an
environment without using NOOR. Participant 1 is diagnosed with glaucoma, but yet, the
glass could not serve his needs. The level of disabilities of patients is not identical, therefore,
two persons with the same case may have different levels and by that may interact with
NOOR in a different way. Perhaps, if NOOR was tested on a larger number of people with
low vision the results may prove that NOOR can serve and help other cases too.

56
 Number of trials Vs. collisions
3

2
collisions

0 y = -0.6x + 2.8
0 1 2 3 R² = 0.9
4 5 6
-1
Trial

Number of Trials Vs. Time

80

60
Time (sec)

40 y = -9.1x + 82.9
R² = 0.93549
20

0
0 1 2 3 4 5 6
Trial

Number of Trials Vs. Velocity

14
Velocity (m/min)

12 y = 1.577x + 4.471
10 R² = 0.89485
8
6
4
2
0
0 1 2 3 4 5 6
Trial

Figure 5.18: a) Number of trials vs. Collisions graph. b) Number of trials vs. Time graph. c) Number of
trials vs. Velocity graph, which belongs to participant 2

57
 Chapter 6

6 Constraints and specifications

6.1 Introduction
This chapter states the various constraints that showed up during the design stage
such as: environmental concerns, ethical, social, economic issues, limitations and
restrictions. Correspondingly, it lists the used components with their specified prices, and
the estimated power dissipation respectively.

6.2 List of constraints

6.2.1 Environmental concerns

Obviously, the design will not cause harms for humans and animals since it is
rechargeable. The surrounding environment may be affected if and only if the Lithium
battery was deposed. Some ignorant people miss the right way of how to get rid of
batteries, which leads to environmental damages caused by the incorrect disposal method,
these materials infiltrate to the soil and underground water, subsequently harm the plants,
animals, and humans. Therefore, the ideal method is to use rechargeable systems.

6.2.2 Ethical and social issues

Releasing the product in the market will help the society in different ways, for example:

• The product will help a huge number of people with low vision, and cut off their
permanent social issues.
• The product will improve the level of communication between the user and others
surrounding them.

Although the idea of the product is new in the Arabian Gulf area, in the belief, people
in UAE will accept it no matter how the glass will look like, since this product has great
humanitarian aspects.

6.2.3 Economic issues

While choosing the design hardware and components, cost has been taken into
considerations; it has been minimized as much as possible to make the product available for

58
almost everyone. The body of NOOR with the camera and the displays costs around 4600
AED.

There are different kinds of assisting aids to help the visually impaired, but yet the smart
glass is not available for the Arabian Gulf area, therefore, our design will be the first of its
kind to be produced here.

The product will provide a new kind of assistance to those who retain low vision, they
will be more comfortable with the product since using the smart glass will terminate the
cane or any other assistance aids.

6.2.4 Limitations and restrictions

Every product must have limitations and restrictions that limit the effectiveness of it,
and NOOR also has limitations such as:

• The proposed design is not applied for all low-vision cases, only the ones mentioned
in the medical terminology section.
• The proposed design does not detect transparent objects since that the laser beam
goes through it and does not have a reflection that can be caught.
• The proposed design does not detect objects that are at a distance more than 15m.
• If a zoom lens has been applied, the resolution decreases.
• The project does not operate in dark environments.

6.3 List of specifications

6.3.1 Parts list and estimated budget

Table 6.1: Parts List and Estimated Budget

Part Name Quantity Supplier Order Place Price (AED)

ZED stereo camera 1 Stereo labs USA 2200

Adafruit display 1 Adafruit USA 410.38

Battery 1 Star Hop Dubai 420

Jetson TK1 1 NVIDIA US 785.38

Body 1 3D printing Dubai 1350

TOTAL COST (AED)

4755

59
 Table 6.2:Power dessipation

Estimated power
Parts
dissipation

ZED stereo camera 380mA x 5V = 1900 mW

Adafruit display 250mA x 5V = 1250 mW

Jetson TK1 2.2A x 12V = 24.6 W

Total Power Dissipation
29.55 W

60
 Chapter 7

7 Conclusion
7.1 Thesis summary
As has been noted, the majority of the blind people around the world return low-
vision. This thesis aims to discuss a procedure that provides assistance to low-vision people
while moving around an unknown environment using the wearable obstacle detection
smart glass "NOOR". The original scene of the surrounding environment is caught by a
stereovision camera, which passes through the processing stage that happens in Jetson TK1
with a co-operation with ZED camera. In this stage, a depth map is constructed from the
original scene using the pixel triangulation method. The luminance, brightness, sharpness,
edge detection and other enhancements have been moderated to fit the medical condition
needs. The Jetsons’s outcome video signal is carried out to the TFT display, which plays the
role of the glasses’ lens.

To ensure that NOOR functions well two studies have been performed. The first
study included healthy people with normal vision to prove that the user needs time to be
familiar with the glass and the rate of learning for each user differs. Study 1 also proved that
by using NOOR, the user would be able to differentiate the closer obstacles and therefore
avoid them. Study 2 included people with low vision, and the results showed that the depth
map representation is very helpful for some of them, and with NOOR they were able to get
rid of the cane or any assistance aids.

Generally speaking, the substantial purpose of this project is to promote the life’s
different aspects, and to improve the ability to avoid obstacles. It is important to realize that
the glass’s function is to consolidate the scene, not to recover the vision.

7.2 Difficulties
As any other implementation process, NOOR’s execution procedure phase faced many
pitfalls that can be summarized as follows:
1. The project’s interest of area tends to computer engineering field more than
Electrical and Electronics engineering field, and thus, setting up the environment
obtained the biggest effort.
2. The main part of the project is to present the scene as a depth map, this process do
not occur easily, several codes and several techniques were tested to ensure a good
quality for the user.

61
 3. The noticed delay between reality and virtual scenes took time to be reduced to an
unnoticed delay concerning that the project works in real time, and the changes
made in the environment should be reflected as fast as possible to the user.
4. JetsonTK1 is a new product and it is difficult to find a suitable display that can be
attached to it and fits the requirements.

7.3 Future work

For the purpose of developing NOOR, several things can be implemented in order to
improve the users’ lifestyle, communication and dependency skills.
• Adding a navigation system will be very helpful for the users especially if programmed
with a guiding voice.
• Indoor maps navigation will help the users to move more freely and reach their
destinations in more accurate way.
• Face recognition system can be added, where the user will be able to identify the
surrounding community, hence, developing their communication skills.

62
 References

[1] B. Mandracchia, A. Finizo, P. Ferraro. Cataract vision mimicked by means of protein denaturation In
egg albumen. Institute of Applied Sciences & Intelligent Systems, Naples, Italy.
[2] Usskamp V, Duebel J, et al. (2010). Genetic Reactivation of Cone Photoreceptors Restores Visual
Responses in Retinitis Pigmentosa
[3] stereo vision. USA: William Hood, 2015.
[4] Wikipedia, 'Computer stereo vision', 2015. [Online]. Available:
https://en.wikipedia.org/wiki/Computer_stereo_vision. [Accessed: 25- Nov- 2015].
[5] Wikipedia, 'Depth map', 2015. [Online]. Available: https://en.wikipedia.org/wiki/Depth_map.
[Accessed: 27- Nov- 2015].
[6] I-art3d.com, 'About Depth Map', 2015. [Online]. Available: http://www.i-
art3d.com/Eng/About_Depth.htm. [Accessed: 26- Nov- 2015].
[7] A. vorbehalten, Obtaining Depth Information from Stereo Images, 1st ed. Dimbacher: Ensenso and IDS
Imaging Development Systems GmbH, 2012, pp. 2-6.
[8] Docs.opencv.org, 'Depth Map from Stereo Images', 2015. [Online]. Available:
http://docs.opencv.org/master/dd/d53/tutorial_py_depthmap.html#gsc.tab=0. [Accessed: 01- Dec-
2015].
[9] Wikipedia, 'Back-illuminated sensor', 2015. [Online]. Available: https://en.wikipedia.org/wiki/Back-
illuminated_sensor. [Accessed: 26- Nov- 2015].
[10] B. Newman, 'BSI sensor (backside illumination): Advanced technology guide', What Digital Camera,
2015. [Online]. Available: http://www.whatdigitalcamera.com/technology_guides/bsi-sensor-backside-
illumination-advanced-technology-guide-60726. [Accessed: 27- Nov- 2015].
[11] G. J. Farrard, P. F. Kenna, P. Humphries. (2002). "NEW EMBO MEMBer's REVIEW: On the genetics of
retinitis pigmentosa and on mutation-independent approaches to therapeutic intervention". The EMBO
Journal 21 (5): 857–64.
[12] M. Haddrill,( 2015, July). Retinitis Pigmentosa [Online]. Available:
http://www.allaboutvision.com/conditions/retinapigment.htm
[13] “Glaucoma in twenty first century”, in The Glaucoma Book, 1st ed. New York (2010). ch.1, pp.03-21.
[14] National Eye Institution. Facts About Glaucoma[Online]. Available:
https://nei.nih.gov/health/glaucoma/glaucoma_facts
[15] Hodge, W. G., Whitcher, J. P., & Satariano, W. (1994). Risk factors for age-related cataracts.
Epidemiologic reviews, 17(2), 336-346.
[16] Klein, Ronald, and Barbara EK Klein. "Diabetic eye disease." The Lancet350.9072 (1997): 197-204.
[17] Wills Eye Institution. A Patient guide to Diabetic Retinopathy. Available:
https://www.willseye.org/sites/all/files/WE_Service%20Brochure_Diabetic_FINAL%201110_0.pdf
[18] Leibowitz, Howard M., et al. "The Framingham Eye Study monograph: an ophthalmological and
epidemiological study of cataract, glaucoma, diabetic retinopathy, macular degeneration, and visual
acuity in a general population of 2631 adults, 1973-1975." Survey of ophthalmology 24.Suppl (1979): 335-
610.
[19] Bird, A. C., et al. "An international classification and grading system for age-related maculopathy and
age-related macular degeneration." Survey of ophthalmology 39.5 (1995): 367-374.
[20] Diomin V. (2011). Low-cost sensor data processing of autonomous mobile robot guide. Available:
http://robotics.bstu.by/mwiki/images/e/ec/Demin-OWD-2011.pdf
[21] Aabloo, A. (2014). THEIAVISION: ULTRASONIC GLASSES FOR PEOPLE WITH VISUAL IMPAIRMENT.
[Online] Available: http://www.tuit.ut.ee/sites/default/files/tuit/arvutitehnika-thesis-bsc-2014-soo-
sander-text-20140528.pdf
[22] CNET, 'BlindSpot: Smart cane concept looks to future - CNET', 2011. [Online]. Available:
http://www.cnet.com/news/blindspot-smart-cane-concept-looks-to-future/. [Accessed: 18- Sep- 2015].
[23] VICE, 'This Greek Teen Invented a Pair of Smartglasses for the Blind | VICE | United Kingdom', 2015.
[Online]. Available: http://www.vice.com/en_uk/read/angelos-getsis-smartglasses-for-the-blind-876.
[Accessed: 12- Sep- 2015].
[24] NMANewsDirect, Wrist-mounted glove helps blind navigate. 2013. [online] Available:
https://www.youtube.com/watch?v=VNa-7n6APDE

63
[25] Discovertechnologies.com, 'Google Glass for Indoor Navigation', 2014. [Online]. Available:
http://discovertechnologies.com/blog/google-glass-indoor-navigation/. [Accessed: 04- Oct- 2015].
[26] Washington Post, 'Everything you need to know about Google Glass', 2015. [Online]. Available:
https://www.washingtonpost.com/news/the-switch/wp/2014/02/27/everything-you-need-to-know-
about-google-glass/. [Accessed: 04- Oct- 2015].
[27] OrCam, 'OrCam - See for Yourself', 2015. [Online]. Available: http://www.orcam.com/. [Accessed:
27- Nov- 2015].
[28] S. Hicks, I. Wilson, L. Muhammed, J. Worsfold, S. Downes and C. Kennard, 'A Depth-Based Head-
Mounted Visual Display to Aid Navigation in Partially Sighted Individuals',PLOS ONE, vol. 8, no. 7, p.
e67695, 2013
[29] Ndcn.ox.ac.uk, 'Stephen Hicks — NDCN', 2015. [Online]. Available:
http://www.ndcn.ox.ac.uk/team/stephen-hicks. [Accessed: 07- Dec- 2015].
[30] Orbbec3d.com, 'Products | Orbbec', 2015. [Online]. Available: https://orbbec3d.com/products/.
[Accessed: 06- Oct- 2015].
[31] Indiegogo, 'CLICK HERE to support Orbbec Persee: World's First 3D Camera-Computer', 2015.
[Online]. Available: https://www.indiegogo.com/projects/orbbec-persee-world-s-first-3d-camera-
computer#/. [Accessed: 06- Oct- 2015].
[32] Wikipedia, 'Kinect', 2015. [Online]. Available: https://en.wikipedia.org/wiki/Kinect. [Accessed: 06-
Oct- 2015].
[33] Zhang, Z. (2012). Microsoft kinect sensor and its effect. MultiMedia, IEEE,19(2), 4-10.
[34] Gonzalez-Jorge, H., Riveiro, B., Vazquez-Fernandez, E., Martínez-Sánchez, J., & Arias, P. (2013).
Metrological evaluation of microsoft kinect and asus xtion sensors. Measurments, 46(6), 1800-1806.
[35] Asus.com, 'Xtion PRO LIVE - Overview', 2015. [Online]. Available: https://www.asus.com/us/3D-
Sensor/Xtion_PRO_LIVE/. [Accessed: 09- Oct- 2015].
[36]Amazon.com, 'Amazon.com: Asus Xtion PRO LIVE, 90IW0122-B01UA: Computers & Accessories',
2015. [Online]. Available: http://www.amazon.com/Asus-Xtion-PRO-LIVE-90IW0122-
B01UA/dp/B00CET0ZBO. [Accessed: 09- Nov- 2015].
[37] Stereolabs.com, 'ZED - Stereo Camera for Depth Sensing', 2015. [Online]. Available:
https://www.stereolabs.com/. [Accessed: 17- Nov- 2015].
[34] Oled-info.com, 'OLED introduction and basic OLED information | OLED-Info', 2015. [Online].
Available: http://www.oled-info.com/introduction. [Accessed: 06- Nov- 2015].
[35] w. 1.66 " '1.66 "flexible Oled Lcd Screen/tft Small Lcd Display/for Smart Phone,Smart
Watch,Wearable Watch - Buy Oled Lcd Screen,Flexible Lcd Screen,Tft Small Lcd Display Product on
Alibaba.com', www.alibaba.com, 2015. [Online]. Available: http://www.alibaba.com/product-detail/1-
66-flexible-OLED-lcd-screen_60044435510.html?spm=a2700.7724857.29.53.UHwlUu. [Accessed: 06-
Nov- 2015].
[36] 2015. [Online]. Available:
http://file:///C:/Users/User/Downloads/developing%20file%20%20NVK176SC001G-C-
0000%20for1.66%20inch.pdf. [Accessed: 06- Nov- 2015].
[37] Appliancedesign.com, 'Selecting the Best LCD Display for Your Product: OLED vs TFT', 2012. [Online].
Available: http://www.appliancedesign.com/articles/93121-selecting-the-best-lcd-display-for-your-
product-oled-vs-tft. [Accessed: 09- Nov- 2015].
[38] Amgoo.com, 'LCD Or OLED? What Is The Difference Between Smartphone Screen Types?', 2015.
[Online]. Available: http://www.amgoo.com/smartphone-blog/lcd-or-oled-what-is-the-difference-
between-smartphone-screen-types. [Accessed: 14- Nov- 2015].
[39] www.alibaba.com, '2 Inch Tft Lcd Module', 2015. [Online]. Available:
http://www.alibaba.com/product-detail/resolution-480-240-with-RGB-
interface_60374800113.html?spm=a2700.7724857.29.240.NQ5a6e. [Accessed: 19- Nov- 2015].
[40] Ebay.com, 'Electronics, Cars, Fashion, Collectibles, Coupons and More | eBay', 2015. [Online].
Available: http://www.ebay.com. [Accessed: 20- Nov- 2015].
[38] Oled-info.com, 'OLED introduction and basic OLED information | OLED-Info', 2015. [Online].
Available: http://www.oled-info.com/introduction. [Accessed: 06- Nov- 2015].
[39] w. 1.66 " '1.66 "flexible Oled Lcd Screen/tft Small Lcd Display/for Smart Phone,Smart
Watch,Wearable Watch - Buy Oled Lcd Screen,Flexible Lcd Screen,Tft Small Lcd Display Product on
Alibaba.com', www.alibaba.com, 2015. [Online]. Available: http://www.alibaba.com/product-detail/1-
66-flexible-OLED-lcd-screen_60044435510.html?spm=a2700.7724857.29.53.UHwlUu. [Accessed: 06-
Nov- 2015].

64
[40] 2015. [Online]. Available:
http://file:///C:/Users/User/Downloads/developing%20file%20%20NVK176SC001G-C-
0000%20for1.66%20inch.pdf. [Accessed: 06- Nov- 2015].
[41] Appliancedesign.com, 'Selecting the Best LCD Display for Your Product: OLED vs TFT', 2012. [Online].
Available: http://www.appliancedesign.com/articles/93121-selecting-the-best-lcd-display-for-your-
product-oled-vs-tft. [Accessed: 09- Nov- 2015].
[42] Amgoo.com, 'LCD Or OLED? What Is The Difference Between Smartphone Screen Types?', 2015.
[Online]. Available: http://www.amgoo.com/smartphone-blog/lcd-or-oled-what-is-the-difference-
between-smartphone-screen-types. [Accessed: 14- Nov- 2015].
[43] www.alibaba.com, '2 Inch Tft Lcd Module', 2015. [Online]. Available:
http://www.alibaba.com/product-detail/resolution-480-240-with-RGB-
interface_60374800113.html?spm=a2700.7724857.29.240.NQ5a6e. [Accessed: 19- Nov- 2015].
[44] Ebay.com, 'Electronics, Cars, Fashion, Collectibles, Coupons and More | eBay', 2015. [Online].
Available: http://www.ebay.com. [Accessed: 20- Nov- 2015].

65
 Appendices:

1- System setup
2- LCD datasheet
3- Jetson TK1 datasheet

66


System Setup

67


LCD datasheet

68


Jetson TK1 datasheet

69