ARIE: Guide Yourself,

Augmented reality based instructions now accessible for

non-programmer instructional designers
By
Gady Yosef and Shachar Oz
Submitted to the Department of Technologies for Learning
in Partial Fulfillment of the Requirements for the Degree of
Bachelors of Arts in Technologies for Learning
at the
HOLON INSTITUTE OF TECHNOLOGY
January 2010
2010 Gady Yosef & Shachar Oz. All rights reserved.
The authors hereby grants to HIT permission to reproduce and to distribute publicly
paper and electronic copies of this thesis document in whole or in part in any medium
now known or hereafter created.

Author .....
Department of Technologies for Learning
January 10, 2010
Certified by.......
Miki Ronen
Professor of Technologies for Learning
Head of Technologies for Learning Department
Accepted by.......
Dan Cohen Vaks
Doctor of Computer Science,
Department Technologies for Learning
1

ARIE: Guide Yourself,

Augmented reality based instructions now accessible
for non-programmer instructional designers
By
Gady Yosef and Shachar Oz
Submitted to the Department of Technologies for Learning
in Partial Fulfillment of the Requirements for the Degree of
Bachelors of Arts in Technologies for Learning

Abstract
Advancements in computer vision and object recognition in real time video, side by side to a
performance boost in computing power and graphics acceleration have allowed rendering virtual
3D objects on top of real time video. This technology, called augmented reality, can be highly useful
in the field of instructional design. Wearing special eye-wear with semi-transparent screens, can
project 3D virtual objects, which might be tooltips, driving instructions or even operation hints, just
as it would have been presented in a products manual booklet. The main advantage is that the
required assistance is presented in real time, exactly when needed, on top of the real product and in
context without forcing the user to look away from the task at hand.
Under these guidelines ARIE system was developed. It is capable of reading and generating
instruction plans using the augmented reality technology. Using ARIE, any instructional designer is
capable of creating a standalone, real time, step-by-step trainings that can be projected on top of an
appliance, while working with it. ARIE saves time and money for organizations that need to train
workers with performing simple step-by-step tasks. With the right instructional design, any worker
could operate an appliance that is new to them without prior knowledge.
This paper describes previous work in the field of augmented reality instructions, ARIEs system
design and architecture, and also shows results from an initial user testing session.

Tags: augmented reality, instructional design, interaction, user experience

Thesis supervisor: M. Ronen

Title: Professor of Technologies for Learning
3

Acknowledgements
Great thanks for BMW for the wonderful Youtube clip that ignited our imagination with augmented
reality technology (http://www.youtube.com/watch?v=P9KPJlA5yds). The clip created the passion
to investigate this technology and its capabilities.
We would like to thank Ori Inbar for showing us the entrance to the rabbit hole of augmented
reality.
And Moti Menachem for pushing us to create a live working system and not be satisfied with just a
concept for one.
For those who made this technology possible for use: Saqoosha and their FlarToolkit, and Eric
Socolofsky for FlarManager. There are many more but they got us going.
And finally, for our supervisors: Professor Miki Ronen and Dan Cohen Vaks, whom without their
redirection we probably would have found ourselves doing a pretty different project by now.

Contents
List Of Figures .................................................................................................................................... 6
1. Introduction .................................................................................................................................... 7
1.1 A Word On Augmented Reality ................................................................................. 7
1.1.1 How Does It Work? .......................................................................................................................... 7

1.2 ARI, Innovative Solution ......................................................................................... 8

1.2.1 Prior Work In ARI ............................................................................................................................ 9
1.2.2 Challenges In Developing ARI....................................................................................................... 11

2. Project Goals And The Design Challenge .................................................................................. 13

2.1 Concept: Generic System To Show ARI On A Single Appliance .....................................13
2.1.1 Real Time Operation Training ........................................................................................................ 13
2.1.2 Hands-Free User Interface .............................................................................................................. 14
2.1.3 User Customization ........................................................................................................................ 14

2.2 System Characteristics And Design ..........................................................................15

2.2.1 Viewer (End User Side) .................................................................................................................. 15
2.2.2 Application Screens ........................................................................................................................ 16
2.2.3 Customization Settings ................................................................................................................... 18
2.2.4 Editor: Tool For Instructional Designer ......................................................................................... 20
2.2.5 Clarifying The Difference Between Animation And Interaction ................................................... 26

2.3 Hands-Free User Interfaces .....................................................................................27

2.3.1 Voice Commands ........................................................................................................................... 27
2.3.2 Body Gestures ................................................................................................................................ 27
2.3.3 Eye Wear Sensors ........................................................................................................................... 28
2.3.4 Multi-Modality ............................................................................................................................... 28

2.4 Information Architecture ........................................................................................29

2.4.1 System Objects ............................................................................................................................... 29
2.4.2 Database Design ............................................................................................................................. 30
2.4.3 Describing Main Procedures And Algorithms At The Viewer Segment ........................................ 32

3. User Test And Research ............................................................................................................... 34

4. Conclusion..................................................................................................................................... 35
4.1 Recommendations .................................................................................................35
4.2 Future Work .........................................................................................................35
4.3 Self-Critics ..........................................................................................................36
5. Bibliography ................................................................................................................................. 37

List of Figures
Figure 1- How Augmented Reality technology works......................................................................... 8
Figure 2- ARI to assemble and operate an appliance (Nilsson & Johansson, 2006) ........................... 9
Figure 3- Collaborative ARI (Zhong et al, 2003)............................................................................... 10
Figure 4- Generic ARI (Yuan et al, 2005) .......................................................................................... 10
Figure 5- Three dimensional augmented work environment (Schwald & Laval, 2003) ................... 11
Figure 6- Head-mounted display; iWrap eye wear by Vuzix ............................................................. 11
Figure 7- QR code, Marker from Zhong et al (2003) and a basic Marker ......................................... 12
Figure 8- Real object as the Marker (CamSpace, 2009) .................................................................... 12
Figure 9- User uses ARIE to activate a microwave ........................................................................... 14
Figure 10- idle state............................................................................................................................ 16
Figure 11- start engagement ............................................................................................................... 16
Figure 12- main menu ........................................................................................................................ 17
Figure 13- secondary menu ................................................................................................................ 17
Figure 14- instructional process ......................................................................................................... 18
Figure 15- help screen ........................................................................................................................ 18
Figure 16- products list ...................................................................................................................... 20
Figure 17- creating a new product ..................................................................................................... 20
Figure 18- activities (two levels menu).............................................................................................. 21
Figure 19- editing a step..................................................................................................................... 22
Figure 20- ARIE augmentation library .............................................................................................. 23
Figure 21- advanced augmentation settings ....................................................................................... 23
Figure 22- positioning augmentation using mouse ............................................................................ 24
Figure 23- positioning augmentation using eye wear ........................................................................ 24
Figure 24- augmentation controlled with marker ring ....................................................................... 24
Figure 25- positioning 2D video with marker ring ............................................................................ 25
Figure 26- accelerometer ................................................................................................................... 27
Figure 27- Marker ring (Nilsson & Johansson 2006) ........................................................................ 28
Figure 28- touch slider (Phidgets) ...................................................................................................... 28
Figure 29- common work procedure of a user with the Viewer ........................................................ 32
Figure 30- Vuzix VR-920, the hardware used with ARIE ................................................................. 34

1. Introduction
The most acceptable method to teach another person to operate an appliance is through simulating
reality as close as possible, as shown in the learning pyramid of Dale (Dale, 1969). A written verbal
or visual explanation, live demonstration by an instructor or animation all have the potential to
support the learning process, but real learning happens when the leaner experiences for themselves
at first hand.
This insight brought instructional designers to create two types of products to train a technical
worker: simulators (or any type of software that simulates the end appliance) and written manuals,
which explain the product with step-by-step instructions.
The manual creates a common language between the appliance and a visual representation of it, and
contains thorough explanations about procedures the appliance offers. The effectiveness of this
method is still questionable can the learner master the appliance just by reading the manual? Will
the manual be available for the learner while on the job, and if so, does it offer an easy guide
through discovering a solution to the issue at hand?
The simulator, on the other hand, faces these challenges by placing the learner right in front of the
system. Its user interface strives to be a complete imitation of the real appliance in its look, feel and
feedback. The main advantage of this method is that the user cannot damage the real product. Its
disadvantage is that the simulator will never be the real appliance and there will always be
differences.
Technological advancement in the past decade have enabled augmented reality, which offers placing
three dimensional virtual graphics on top of the real world. The teaching potential of this
technology continues to gather support among instructional designers around the world. Since this
is still an emergent technology, previous work were focused on a limited context or showed narrow
customization or reusability.
This work presents a highly generic system that offers the creation of augmented instructions that
will assist the user operating an appliance. Its user interface doesnt require any coding skills thus
allowing common instructional designers to do their work without troubling themselves with
technicality.

1.1 A word on augmented reality

Azuma defined a system as augmented reality if it follows these three principles: combines the real
world and the virtual one; offers interaction with the virtual world in real time; works in 3D
(Azuma, 1997; Azuma et al, 2001). This definition is accounted as a starting point for all augmented
reality development. The most well-known example of this technology might be the pilots helmet
that presents various data in real time and extends the pilots skill and knowledge.

1.1.1 How does it work?

This technology demands specific hardware to work. A video camera captures the reality, and the
information is transferred to a computing device. Using image processing the computer adds the
augmentation to the video. The result is transferred back to the users screen, or head mounted
7

display.
To be able to do that, we need a reference virtual plane which can be referred when positioning new
objects. To create this plane and keep tracking it, several computer vision solutions have been
developed (Figure 1).
ARToolKit, one of these solutions, was invented at 1999 (Zhong et al, 2003), and it allows identify
and track black bordered squares that have a pre-defined graphics in its center. These squares are
called Markers. The graphic drawing in the center of the square is changeable, as long as it stands
several rules, such as the shape must be asymmetric.

Figure 1- How Augmented Reality technology works

1.2 ARI, innovative solution

Training using augmented reality (ARI: augmented reality instructions) is an innovative method to
deal with common problems the instructional designers face, such as: training effectiveness and
availability of content after a training course. This chapter will show how ARI can become a
solution to these issues, its advantages and drawbacks.
The augmented reality user is forced to interact with real object in a real world. Thus, learning and
training can happen in front of the appliance itself. In addition, ARI doesnt require prior technical
knowledge before first operation. Schwald & Laval (2003) found that ARI is more effective than
simulators due to the fact that it is closer to the real work environment while keeping low risk of
damaging the appliance. Yuan et al (2005) compared ARI to written manuals, and found that while
with simple procedures there was no significant difference, in complex ones ARI was much more
effective.
Therefore, ARI actually saves time, money and personnel to the organization. After all, the end goal
of any instructional design is to achieve maximum training in minimum instruction time. The
organization considers training time as unproductive, and the instructional designer should be able
to prove that the training will return the time and money invested in it. Using ARI, the training
occurs in front of the appliance, while using it and working with it. Saving training time also cuts
other expenses, such as: training rooms, printed materials, simulators and software, etc. The
organization actually gets a more prepared worker in significantly less time without investing
money in unproductive training.
Even the best of systems might fail if it wasnt well integrated into the organizational workflow.
8

Understanding the accessibility and usability of the system as well as nurturing the feeling of
confidence the user has inside the system, have crucial impact over the assimilation of the new
system in the organization. Research has discovered that ARI has high potential for integration, and
users have good understanding of the technology (Schwald & Laval, 2003; Nilsson & Johansson,
2006). Many testers have had a positive impression on the usage of augmented reality in training
procedures, and all pointed out that they can vision how this technology be part of their professional
future.

1.2.1 Prior work in ARI

ARI actually exists from the end of the 90, and several projects have been developed in the fields
of: health (Nilsson & Johansson, 2006; Vuzix, 2009), industry and factory (Zhong et al, 2003;
Schwald & Laval, 2003), military (Vuzix, 2009; Boeing, 2005), transportation (Reiners et al, 1998).
Projects dealt with different aspects of technical training: assembly (Nilsson & Johansson, 2006;
Yuan et al, 2005; Reiners et al, 1998), operation (Nilsson & Johansson, 2006), maintenance
(Schwald & Laval, 2003) and identifying and fixing malfunctions (Schwald & Laval, 2003; Lipson
et al, 1998).
Nilsson & Johansson (2006) tried ARI on 8 hospital workers with high accessibility to emergent
technologies (but didnt experience augmented reality before). Each tester had two experiments
with ARI. In the first one, they operated a medical appliance and in the other they assembled a
surgery tool (Figure 2). Some testers said that this type of training had significant advantages over
instructor led guidance because the given instructions were very accurate and a lot of time was
saved. But most of the users complained about the lack of human support, such as: asking a quick
question or seeking for advice when feeling uncertainty after performing an action.

Figure 2- ARI to assemble and operate an appliance (Nilsson & Johansson, 2006)
Zhong et al (2003) have developed ARI for wiring a circuit board for ATM machines. The system
was a one-on-one conference call between the instructor and trainee. The trainer could see what the
trainee sees and assist them in real time with visual and auditory explanations (Figure 3). The
instructor could draw and remove graphic augmentations. The main disadvantage of this system is
that it was not in front of a real live product, but rather it was a virtual simulation on a still image.

Figure 3- Collaborative ARI (Zhong et al, 2003)

Yuan et al (2005) developed a generic system to create augmented assembly instructions. The
system also had a unique user interface to operate the system. The main disadvantage of the system
is that it works with no direct connection to the real product. Meaning, the user would watch 3D
augmented animations, but it was not depended with the physical object they had before them
(Figure 4). The menu system of the product was not influenced by the object the user now faces.
Similarly to working with training manuals, the user was responsible to make the matching between
the virtual object and the physical parts in their hands.

Figure 4- Generic ARI (Yuan et al, 2005)

Schwald & Laval (2003) built a system called StarMate in order to train maintenance factory
engineers and assist them during work. The main lesson the research had, was to separate the two
use-cases into two independent systems: one for training and another for real time job assistance.
The training itself was through simulating scenarios on top of a real product. Another important
addition of the research was in the creation of 3D work space (Figure 5). The user was surrounded
with virtual augmentations, each offers different type of assistance.

10

Figure 5- Three dimensional augmented work environment (Schwald & Laval, 2003)
Lipson et al (1998) showcased a web-based system that is used as a maintenance tool in factories.
The worker used the system to diagnose the problem with the appliance, and then the system would
offer step-by-step instructions to fix the problem.

1.2.2 Challenges in developing ARI

Reiners et al (1998) is the earliest research with ARI. They showcased a system that explained how
to assemble a lock mechanism into a vehicle door. Even though almost a decade past, nearly each
project recommends further technological development is needed to turn this technology into a
commercially available product.
One reason for this is that the hardware is still under development. It is too heavy, and
uncomfortable to carry and wear. The hardware should not limit the users freedom of movement in
any way. For example, a head mounted display, which was used so far, is very cumbersome (Figure
6). Products would need much lighter equipment, like the Vuzix eyewear, that was used for this
project (Figure 6). A wireless connected eyewear is probably the next step (Schwald & Laval,
2003).

Figure 6- Head-mounted display; iWrap eye wear by Vuzix

Software is in its early stages too. Tracking performance is continuously being upgraded (Schwald
& Laval, 2003). Zhong et al (2003) suggested an updated marker with better tracking. Every
passing day new tracking technologies are released: whether its 2D black and white (QR-Marker,
2009) or RGB image of real products (CamSpace, 2009).

11

Figure 7- QR code, Marker from Zhong et al (2003) and a basic Marker

Figure 8- Real object as the Marker (CamSpace, 2009)

The need for human interaction during training is another user requirement (Nilsson & Johansson,
2006). We are all familiar with the aversion we feel when electronic answering machines do not
understand us. Even when a trainee receives precise instructions to perform, the user might need
positive feedback for their performance. Using new technology is known to have more potential to
create uncertainty by the user.
Another challenge in developing augmented reality based systems is the user interface and user
interaction with the product and information. Despite the huge amount of research in that field in
the past few years, the subject is still in its beginning. Almost every paper focusing on the
interaction tried a new path. Nilsson & Johansson (2006) asked the user to assemble a small object,
so the tester was asked to place a ring on their finger. When the ring was identified and tracked, an
assistive 3D representation was attached on top of it. Researchers used Markers in order to allow
selection: users had to touch a selected object with the Marker (Nilsson & Johansson, 2006;
Zhong et al, 2003). To prevent unintentional selections, a confirmation popup jumped.
As mentioned above, the majority of the projects done so far were very narrow in terms of its goals
and meant to showcase the potential of the technology (Zhong et al, 2003; Schwald & Laval, 2003;
Lipson et al, 1998; Reiners et al, 1998). However, a system developed for very specific use cases or
content is unnecessarily transferable to other fields.

12

2. Project goals and the design challenge

Previous ARI work showed great potential augmented reality has in the field of training. Most of
the testers themselves have grasped that potential, even if this was their first encounter with the
technology (Nilsson & Johansson, 2006; Schwald & Laval, 2003).
The main goal of ARIE project is to make that potential more accessible to instructional designers
for daily use. For the majority of the previous attempts and ARI work was made by programmers
and technologists, since they were the only people who could deal with the technical issues. We
would like to transfer the reins of such a powerful tool to the instructional designers, rather keeping
it within the programmers control.
Therefore, the system has to be able to create training sequences for different types of appliances,
using a unified generic route. The system has to adapt to a wide audience, languages and
disabilities, and offer various types of media. The system should be highly customized while
keeping a simple and intuitive user interface. This is another challenge because usually generic
systems are highly complex in its user interface.
ARIE integrates several lessons from past work to become more effective. ARIE offers the generic
creation of a step-by-step instructions (as shown in Yuan et al, 2005), acts only in front of the real
appliance (as opposed to Zhong et al, 2003) while giving clear instructions with simple steps to
reproduce (as in Schwald & Laval, 2003). ARIE unifies the training with real time work on the real
appliance, thus saving time and money building two separate systems (as Schwald & Laval, 2003
recommended). ARIE offers extended help mechanism that confronts the need of human support
(Nilsson & Johansson, 2006), and all is made with simple user interface and a 3D work
environment (Zhong et al, 2003).
Above all, ARIE presents a significantly innovative way that can change technical trainings. With
this system, any instructional designer in the organization could generate training, even if they do
not have programming skills at all. The trainees are not really wasting precious time on training,
because they are learning and working at the same time.

2.1 Concept: generic system to show ARI on a single appliance

ARIE is capable of showing information regarding a single appliance at a time. The system is best
suited for appliances that are independent and doesnt need other instruments or information in
order to run, that the user has no control of. A future generation of the system could make that leap
as well.
It is recommended to use this system to operate appliances on non-critic systems. Operating a
combat plane for example, demands a short feedback loop and performing several actions in
parallel. These characteristics doesnt match the system capabilities, due to technological issues
with unstable response time and accuracy, but also because we prefer giving the learner the time
they need to learn over preserving the complexity of operating a critical system.
This definition suites many appliances in our home and office environments: whether its the office
photocopy / printer, washing machine, kitchen oven, factory post. Actually any written manual can
be replaced, or enhanced, with an add-on to the described system.

2.1.1 Real time operation training

ARIE is, in a way, an interactive instruction manual in the real world. The users can spend their first
13

stages with an appliance side by side with ARIE, and learn how to use it while really using it. Not
by reading papers, nor by watching videos on a website. The user puts on a unique eye wear, look
towards the appliance and the information will pop up, on top of the appliance itself.

Figure 9- User uses ARIE to activate a microwave

Using ARIE saves the need of a mediator between the learner and the learning process. The entire
work is performed on the real product and in real time, with no prerequisites. The learning period
merges with the on-boarding stage of a new worker. The worker becomes productive even during
his training period. A worker will be able to approach appliances that they did not know before.
Each training task is divided into simple smaller steps, which are clear and achievable by any user.
The system is an easy platform to create training sequences, but the responsibility for the
effectiveness of instructions stays in the hands of the designer.

2.1.2 Hands-Free user interface

Since the goal of the system is to allow the user to freely operate the appliance, the control of the
system itself must be hands free. The user should not be forced to look away from the appliance or
walk away from it.
Therefore the system must offer interactions that will allow progressing in the learning sequence
with no interference. Technological solutions, such as voice control, motion sensing and gesture
control can be optional here. See relevant chapter further on.

2.1.3 User customization

To create an effective learning sequence we must make sure the learner receives full attention. Each
learner has different preferences and needs, starting from colors, fonts, brightness, training speed
and also instructional content, such as: preferred help tools, voice explanations, hints, etc.
ARIE lets the user the freedom to decide their preferred type of media they feel more comfortable
with. The user asks the system to provide extra help regarding the performance of a specific step in
a format they choose: textual, auditory, visual (3D model, image, animation). This would be
available as long as the instructional designer created that variety of content.
The user feeling of control in the system have been translated into these product requirements:
1. Open step-by-step training sequence: the user can skip and move freely between steps.
2. Control on the amount of information presented at once.
14

3. Control of type of information presented: animation, 3D model, video, images, text, voice.
4. Control of the timing: user decides when a new step will be presented or when to show more
assistance.
5. Visual preferences: color, font size and type, location of help screen and menus, animation
speed, scale of a 3D model.
6. Learning the users preferences and adjusting future trainings according to it.
For the instructional designer the system should allow full training management:
1. Control of all menus and steps in a product.
2. Editing, duplicating and management of activities.
3. The system should contain an assets library of media objects, both 2D and 3D, sounds and
texts. This should be sufficient to create most instructions. The designer can extend the
library and add objects of their own.
4. Designer could build an activity in front of the appliance itself in real time, as well as in
front of a personal computer, without the product close by.

2.2 System characteristics and design

2.2.1 Viewer (end user side)
Standard use case: operating a washing machine
We have all been there. Standing in front of our washing machine, wishing for a colored laundry,
troubling in remember where to put the cup, how much to put and what value to turn the switches
to. So instead of googling the process or searching the manual, one could use the system designed
here.
User puts on the eye wear, starts the software and stands in front of the machine (Figure 10). The
system recognizes the washing machines model and presents the activity menu allowed with that
machine, for example: wool laundry, colored laundry and white laundry (Figure 11). The user
selects the chosen activity using a voice command, such as: Wash Colored Clothes (Figure 12). A
virtual 3D arrow would pop up and point to the place you should dump the powder into (Figure 13).
A simple voice command, like Go on or Next step, tells the system to show the next step. A
circled sign shows above the degrees button, telling the user to set it to 40 degrees. If the user is not
so sure what to do exactly, they simply ask for Help Please. The simplified circled arrow would
be replaced with a 3D model of the button, animated turning to the right direction and stopping at
40 degrees. A voice over would explain the task as well.
In this way, the process would continue until the task is complete. Taking off the eye wear shuts
down the application.
This idea can also be used for office uses, like activating an office printer or a factory post. Using
this system any training period will become significantly shorter, assuming the sequence is built
correctly and is understandable to a new user. The only thing the user should know, is what they
wish to perform.
15

2.2.2 Application screens

User sees the world as it is (Figure 10). The application identifies the markers that are attached to
the different appliances.

Figure 10- idle state

After enabling the application, it will bring the main action menu of the identified marker (Figure
11). The marker itself would glow in order to differentiate between several markers. The menu itself
would show the name of the product and model, and expand the actions offered with it. An arrow
would mark the current selected choice. The user selects one of the menu options using voice
commands.

Figure 11- start engagement

16

Figure 12- main menu

User scrolls between available actions using voice commands (scroll down or Scroll up or
show more, etc). Selection is made with a select command or by a full voice command such as
unfreeze pie.

Figure 13- secondary menu

Training now begins (Figure 14). The current step in the sequence would be presented in the top left
part of screen. The current steps augmentations would be shown on top of the real world. List of
the voice commands would be shown in the bottom of screen, such as help, next step or back
to menu.

17

Figure 14- instructional process

When user asks for extra assistance, assuming one has been entered, another augmentation would
come up, according to the definitions of the instructional designer (Figure 15).

Figure 15- help screen

2.2.3 Customization settings

The systems options menu answers personalization challenges of variety of audiences. The options
offer setting the colors, sound and volume, languages, user interface, fonts, and operating settings.
If the users settings might conflict with an instructional design definition, they will be notified for
it, but their preference would come first.
All user interface is shown over the real world, which is a problematic background in terms of
contrast and color. This is why all augmentations and virtual aid will be semi-transparent in order to
help the user differentiate the real and the virtual. The outline color of the augmentations can be set
from the options menu. There is also the option to let the system select the most contrasted color in
real time.
Options offers controls of the volume of the microphone and speakers. The default language would
be taken from the users operating system, but it is optional to switch it. Selected option will set the
spoken voice and written texts of the app. It will also be optional to use subtitles in order to allow
hearing impaired users to use the software.
18

User will be able to choose their preferred user input method, such as: voice commands and gesture
recognition control. More interfaces are described further on. This feature allows the system to be
accessible for a wide variety of physical disabilities.

19

2.2.4 Editor: tool for instructional designer

Instructional designer required to assemble a new sequence for a specific product enters the system
with their unique username. The system identifies the user and uploads the products they have
created so far, or that they are related to. The user can update, create and delete products (Figure
16).

Figure 16- products list

When creating a new product, the designer is asked to add the name, model, name of company and
an image of the marker (or ask for an auto generation of a new marker) (Figure 17).

Figure 17- creating a new product

Each product has categories, activities and steps. Example categories can be: operation and
troubleshooting. Example activity can be: washing colored clothes or washing white clothes
and example step can be open appliance door. Adding new category or activity is made with the
20

buttons in the lower left. The editor allows creating endless menus and items, but it is recommended
to keep the end product with no more than two levels of menu in order to preserve the accessibility
of the product (Figure 18). The designer can always preview the built menus and see it with the
eyes of the end user.
Adding steps is the most important process and highlight of the software. It is essential to remember
that the steps must be simple to perform even by a new user. The way a step is defined and
presented influences directly on the user performance. Therefore it should be defined by a veteran
instructional designer. The designer can edit, delete and rearrange the steps. They will be presented
in the order they are shown here. Each step can be previewed and seen from the eyes of the end
user.

Figure 18- activities (two levels menu)

Adding content for a step is made through another screen (Figure 19). This is where the designer
adds an augmentaion (text, image, video, 3D model, animated 3D model or voice), sets its
preferences and edits ways of interaction with it. ARIE editor offers placing more than one
augmentation, but it is recommended to keep the screen clean, in order to keep the user focused on
the task at hand and not cause conflicting instructions. Rather, use the extra augmentations as extra
help when user asks for any.
user can load an augmentation from the their computer or from a library of ready-made generic ones
(Figure 19, left panel). ARIE is supplied with a library of generic augmentations, in order to allow
instructional designers to start working immediately. This offers working without a 3D artist. The
library allows uploading more content and arranging its directories.

21

Figure 19- editing a step

Each augmentation has advanced settings, where the designer defines if the selected augmentation
will be accounted as extra help (and be presented only if asked for one), or should it be presented
right away. Designer can set an auto start for videos or animations, or make it start upon a users
request.
The designer can also add up to two interactions the augmentation will have with the user. Example
interaction: as the user gets closer to the augmentation, it grows. The designer would have several
variables they could adjust for any interaction. In the example case, designer could adjust the
starting distance, the factor of grow and the final distance.
The right panel (Figure 19) is actually a live camera feed allowing the designer a continuous
preview of the resulted step. When augmentation is loaded, it shows up in that window, and the
designer can reposition it, scale and rotate it uaing the mouse (Figure 22). External view button
offers repositioning the augmentations while the designer stands in front of the appliance with eye
wear equipment. They will have a marker ring to control the position of the augmentation (Figure
23Figure 24).

22

Figure 20- ARIE augmentation library

Figure 21- advanced augmentation settings

23

Figure 22- positioning augmentation using mouse

Figure 23- positioning augmentation using eye wear

Figure 24- augmentation controlled with marker ring

24

Figure 25- positioning 2D video with marker ring

25

2.2.5 Clarifying the difference between animation and interaction

This short chapter showcases several use cases of the augmented reality technology for training
goals and tasks. We call these interactions, as they are a form of interaction between the user and
content using the marker.
Training goal

The users action

Relevant example or use case

Exploring inner parts or

layers of an object or
appliance

Moving on the Z axis of the

marker (closer and further), will
show different layers of the
object.

Learning how a clock or a safe

works.
How does the human body
works?

When the user is far away, they

see different information
compared to standing close to
the marker.
Exploring inner parts or
layers of an object or
appliance

When an object is placed inside

a pre-defined location, a special
animation occurs.

Only when user brings the fuel

funnel above the right place in the
car engine, a virtual animation
will show its ok to pour it in.

Object can change colors

Angle or distance from marker

changes a color from a defined
palette

Warm-cold; closed-open; rightwrong; state in a short process;

Visual object

User action and its influence

on the visual object

Relevant example or use case

Emphasis element of
some kind (square,
marker)

Positioned according to the

marker

Attaching a visual emphasis to a

specific point in the appliance

Arrow

Simple animations: moving

towards a specific direction or
turning around a pivot

Showing a direction, pointing to a

specific spot or where to press

Info bubble (text/ image/

drawing/ video) with
more emphasis

An important instructional step,

like pressing in two places at a
time

Two actions in parallel when the

user cannot see both locations on
the immediate sight. Like: press a
button and push the door at the
same time.

Button

Pressing animation that shows a

special way to press it

Illustrating the position of the

button and the way it needs to be
pressed to be activated

Slider

Moving a marker on x or y axes

will control the sliders value

26

2.3 Hands-free user interfaces

Schwald & Laval (2003) found that users dont necessarily refer the mouse and keyboard as
effective tools to position object in a 3D environment (66% to the keyboard; 72% for mouse).
Clicking mouse buttons is not optional when the user should have their hands free to control the
appliance.
The most basic functionality required to control any software is: selection, canceling (undo) and
scrolling between available actions. Effective user interface allows easy scrolling between the
available actions on screen. Intuitive interface is one that helps the user to easily grasp the actions
they are required to make, without long tutorials.
This chapter will survey several effective and intuitive hands free interfaces. The detailed
description also includes uncomfortable ways of interaction, and our considerations in choosing the
interface for ARIE.

2.3.1 Voice commands

Voice control activation is fully functional in the Windows Vista operating system, and is intended
to be used by physically disabled. The activation is made using pre-defined commands, such as:
run this, next page, exit, etc. This interface is intuitive because the computer can
understand the user and their will. It is highly important to choose commands that the user would
really use and be comfortable to say. Microsoft has a voice commands SDK ready for use.

2.3.2 Body gestures

Project NATAL (Microsoft, 2009) showcases the future of body gesture interfaces. Using this
interface has the potential to detect head movements and let it scroll between items in a menu, and
selection of the wanted item will be made with a head nod.
Project NATAL works using computer vision, but the described interface can also be developed
using accelerometer sensors, which can detect the slightest movement. These sensor exist in almost
any new cellular phone. Accelerometer (Figure 26) detects movement speed on all axes (x,y,z).
Inside the eyewear used in this project, Vuzix VR-920, there are three accelerometers.

Figure 26- accelerometer

Another idea is that the index finger would replace the mouse pointer. Scrolling through actions
would be made by pointing the options on screen or by swiping through options. Selecting an action
will be made with a tap/ press gesture. This type of interface can be developed in several ways:
27

computer vision can identify and track the index finger or by attaching a marker to a ring (Figure
27).

Figure 27- Marker ring (Nilsson & Johansson 2006)

2.3.3 Eye wear sensors

Attaching soft buttons or other sensors on top of the eyewear can also generate an interesting
interface, but it will be less intuitive than previously discussed methods. This is rather technically
easy to attach a touch slider, where a swipe of finger scrolls between options and tapping it selects
the current option (Figure 28). Another technic is to have two buttons that will scroll and select.

Figure 28- touch slider (Phidgets)

2.3.4 Multi-modality
At the end of the day, a user that moves their head to control a software will look weird, and a user
that will operate their computer using voice commands will find themselves static. The solution to
this issue is using several methods of interaction, or what is called: multi-modality. For example:
the user uses their finger to scroll between menu options and selects with a voice command.

28

2.4 Information architecture

2.4.1 System objects
Physical objects
Appliance/ product: the object which the user wishes to operate.
Marker: a printed content on paper that can be identified by unique software. Using the marker, the
software creates a 3D reference plane that on top of it we can place 3D virtual objects.
Training objects
Step: simple and performable instruction.
Activity/ training sequence: a sequence of steps that together make a significant and identifiable
change in the appliance.
Help/ assistance: a different way to explain a step using other media.
Super-activity: several activities around the same type of actions. Example: there are several types
of laundry: white, colored, wool.
Category: several sub menus and activities that represent a way of using the product.
Programming objects
Augmentation: any virtual object added on top of the real world. Can be any media: text, animation,
3D model, image, video or sound.
Material: allows transforming any media type into an augmentation.
Other basic concepts
Marker-based object: augmentation that is attached and linked to a marker. Example: an arrow that
is attached to the appliance door.
Screen-based object: augmentation that is detached from the marker. Example: menu or explanation
bubble that are constantly show on screen no matter where the user looks at.
Animation: continuous change in the augmentation that is happening with no controls of the user,
such as: moving pointing arrow.
Interaction: change in the augmentation that occurs due to a users action, such as getting closer to
an augmentation enlarges it.

29

2.4.2 Database design

Xml file structure
<products>
<product id="1" name="microwave">
<category id="1" name="cooking">
<activity id="1" name="potatoes" UItype="voiceCommands">
<steps>
<step id="1" name="pin potatoe">
<object id="0" media="3d" source="filename.dae" alwaysShow="true"
xpos="0" ypos="0" zpos="0" rotationx="0" rotationy="0"
rotationz="0" scale="10" />
<helps>
<object id="1" media="image" source="filename .jpg"
alwaysShow="false" xpos="0" ypos="0" zpos="0" rotationx="0"
rotationy="0" rotationz="0" scale="1" />
<object id="2" media="text" source="text text" alwaysShow="false"
xpos="0" ypos="0" zpos="0" rotationx="0" rotationy="0"
rotationz="0" scale="1" />
</helps>
</step>
</steps>
</activity>
</category>
</product>
</products>

The product is the most basic element in the system. One product can have large number of
activities (instructional sequences). Any activity is a set of steps, and each step can have several
visuals and helps, that are shown to the user. Here are the characteristics and functionality of each
of these features.
<product id="1" name="microwave">

Object name

Product

Name attribute

The name of the product that is shown to the user (Figure 11)

<activity id="1" name="potatoes" UItype="voiceCommands">

Object name

Activity

Name attribute

The name of the activity that is shown to the user (Figure 12)

UItype attribute

The type of user interface of the current activity. Designer can select one
interface from a closed list available, or code new ones. Ready-made user
interfaces are: voice command and Wii remote.

30

<step id="1" name="open door">

Object name

Step

Name attribute

The name of the step that is shown to the user (Figure 14)

<object id="0" media="3d" source="filename.dae" alwaysShow="true" xpos="0" ypos="0"

zpos="0" rotationx="0" rotationy="0" rotationz="0" scale="10" />

Object name

Object (Media)

Media attribute

The type of presented media: voice, 3dShape, 3dStatic, 3dAnimation,

image, text, video

Source attribute

Name of file that is supposed to be presented. For Text media this would
contain the content value of text.

xpos , ypos , zpos

attributes

The position offset from the center of the marker. If these properties will
not be defined or valued 0, the object would be presented on top of the
marker.
2D media is shown on a dedicated window and therefore ignores this
attribute.

rotationx, rotationy,
rotationz attributes

Rotation on the different axes

Scale attribute

Affects only 3D media

AlwaysShow
attribute

Boolean variable that defines if the media would be kept throughout the
entire step, or disappear when next help is presented.

31

2.4.3 Describing main procedures and algorithms at the Viewer segment

Figure 29- common work procedure of a user with the Viewer

1. Software initialization:
a. Upload array of marker to identify the different products.
b. Empty the stage from leftover objects.
c. Zero all variables.
d. Setting the variable Activated or Deactivated for the software. Identifying known markers
runs in the background anyway.
2. When marker is identified:
a. Outline the marker with a blue line. Play identify sound effect.
b. From now on, the rest of the procedure will happen only when the system is turned on, the
active state (user sets this on the main menu)
c. Marker is added to the array of identified markers (used in order to keep track of content in
case of a momentary loss of sight of the marker.
d. Start a timer.
e. Is there more than one identified marker?
i) Yes: offer switching between them (with a scrolling interface). Allow selection of one of
them. When a marker is selected, go to 3.
ii) No: wait 2 seconds. No other marker was identified during that time?
(1) Yes: ask the user if they wish to get the activity menu for this product.
3. When selecting a product (as long as the marker is still in the array of identified markers):
a. Is the stage empty from previous procedures on other products AND is there no running
delay (that is activated when an active marker is lost)?
i) Yes:
(1) Show product menu on stage (settings, update breadcrumbs)
(2) Mark stage as not empty. Used to avoid having more than one menu at a time.
(3) Load activity menu from products.xml
32

(4) Generate the menu

(5) Allow scrolling interface between activities
4. When selecting activity (as long as the marker exists in the identified array):
a. Load all necessary steps from the xml
b. Create animations
c. Show the first step
d. Update breadcrumbs
5. When a marker disappears:
a. Is this marker defined as active (meaning this is a marker that is attached to current
running procedures)?
i) Yes:
(1) Set timer for 3 seconds. Is visual contact with marker not renewed?
(a) Yes: remove marker from identified markers array
(b) No: stop timer. Add marker to identified markers array
ii) No: remove this marker from identified markers array
6. Settings of application menu:
a. Change sound volume: of voice over and effects
b. Set colors of all visual elements: set the main color or let the application to select an
automatic color (according to the background).
c. Activate / deactivate of the application. When deactivated, app will identify markers but will
not present activity menus.

33

3. User test and research

ARIE system was first tested on participants of the Synthetic Reality conference at January, 10th
2010. The age range of users was between 22 and 70. First the users received a short briefing about
the idea behind the system and its goal. Some also watched a pre-made presentation. Then they used
the system, when the developers stood nearby in case of need. Although this wasnt an organized
user testing, several essential lessons were learned.
The first and most important point users made was that the hardware is far from perfect. The
eyewear we used was significantly more lightweight compared to an average head-mounted display
(Figure 30), but users found it heavy and bulky. Its center of mass is on the front causing the user to
tilt their head downwards.

Figure 30- Vuzix VR-920, the hardware used with ARIE

Users suggested that augmentations would be semi-transparent in order not to hide the real world.
The transparency also helps the user to differentiate the real from the virtual. This lesson was also
learned at Schwald & Laval (2003).
Using new and unfamiliar technology, users are more careful and cautious before acting. To ease
their on-boarding stage, a human-like escort should be considered. It can be a virtual agent or
simple voice help, in order to guide the beginner.
Some users imagined that the system would perform the tasks automatically for them. This was a
very interesting finding, especially when they knew that it wasnt meant to.
The user interface of the system is a very integral part of it. During the user testing session, users
had to work with a mouse, because the voice commands wasnt developed yet. Using a mouse gave
a false experience.
The sensors field of view must match the one of the real eye (about 50 degrees). The web-camera
used had a field of view that was a little narrow than that, and users felt the difference.
Zhong et al (2003) pointed that the minimal frame rate required to augmented reality is 10 frames
per second. ARIE managed to reach 25 frames per seconds. But apparently there was still a sensible
delay between the camera and the real world.
Tracking quality was not perfect, and users complained on the fact that augmentations sometimes
flickered.
We noticed that nearly all users didnt close the microwave door during the training. After
examining the sequence of instructions we saw that there was no step pointing that out. The lesson
is that the instructions should be as simple as possible and never skip any level, no matter how
obvious it may be.
34

4. Conclusion
During this research we have developed one segment of the ARIE system, the Viewer. The concept
of ARIE design was to create a working environment that the user receives only the information
they need in order to work, and only when they need it. A preliminary research shows that this
message went through and the users have grasps this capability.
Nevertheless, we could see that even though there was a significant improvement in the hardware
over the years, it is still the main limit for a wide use. The Vuzix VR-920 eyewear were heavy,
bulky and cumbersome for the users. The fact that the eyewear were wired directly to the computer
were a noticeable limitation in order to work freely in front of an appliance. Only when the
hardware would be wireless and truly lightweight, then it could have a commercially wide market.
This project was based on the concept of creating a 3D workspace (Schwald & Laval, 2003). Maybe
this was the cause for insecurity and lack of confidence many users felt on first interaction. It is
very important to improve friendliness and intuitiveness of innovative and extreme technologies.

4.1 Recommendations
In addition to the lessons learned from the user testing, we have several more recommendations for
future ARI systems.
1. Make sure the user understands the wanted end result, as best and as clear as possible. This
was also found highly important at Zhong et al (2003).
2. Use voice over to help the user understand the system itself and escort them during the
process.
3. 3D media should be allowed to be either attached to a marker or detached from it, the same
way 2D media is. The Editor should allow that feature. This is useful when instructional
designer wishes to show a process but side by side to the real world instead of hiding it
entirely.
4. The Editors generic animation library should contain also 2D and 3D animations, like: hand
push, pull or grab; button pressed or turned. These types of generic animations would be
highly useful for many types of activities and appliances.

4.2 Future work

There is still development needs to be made on the Editor segment and complete features of the
Viewer, as described above. The preliminary study showed several more aspect needed to be taken
into account.
After the end of the development each of the systems segments needs to go through an extensive
user research, both quantitative and qualitative, in order to test its accessibility and usability. This
research showed that we have to focus on the intuitive aspect of operating the Viewer.
New features we didnt think about that are to be considered. The user should be able to decide
which windows or elements they want to hide (or show) on their screen, i.e.: some users would like
to hide the help window. An important feature for the Editor segment is having a generic animation
databank, for the instructional designer to choose from. Example: an animation of a moving back
and forth can be attached to an arrow in order to make it move in a certain direction.

35

4.3 Self-critics
The project management on the development stage was disorganized, and the system concept
changed several times, and we often had to validate probability and experiment programming.
There was very little time to pay attention to content management and content related issues.
We should have had more time trying out the software with new users, who were not related to the
project. This is needed in order to find issues in the way the system works that we missed. This
could potentially ease the development and the number of software updates you need to do over
time.
Moreover, the way we worked as a team as well as the way we worked with our professors werent
effective. We had ill communication. The project was highly demanding in terms of code
development, and we worked hard towards reaching a working product, without stopping to get
advice or share progress, that might have helped in the development of the end prototype.
On a positive note we can say that we took a very challenging project, jumping into uncharted
territory, both in the academic world as well as in the industry. The augmented reality technology is
still in its early stages of development throughout the world. Despite that, we feel we have reached
a successful product that we believe is a great starting point for continuing the development of the
project until it becomes a real working product.

36

5. Bibliography
Azuma, R., (1997). A survey of Augmented Reality. Presence: Teleoperators and Virtual
Environments. 6:4 (1997) 355-385 http://www.cs.unc.edu/~azuma/ARpresence.pdf
Azuma R., Bailot, Y., Behringer, R., Feiner, S., Simon, J. & Macintyre, B., (2001). Recent Advances
in Augmented Reality, IEEE Computer Graphics and Applications. Nov/ Dec 2001, 34-47
CamSpace Website, 2009. http://www.camspace.com/
Dale, E. (1969). Audio-Visual Methods in Teaching. New-York: International Thomson Publishing.
Lipson, H., Shpitalni, M., Kimura, F. & Goncharenko, I. (1998). Online Product Maintenance by
Web-Based Augmented Reality. Proceedings of CIRP Design Seminar on New Tools and
Workflows for Product Development.
http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.21.6290&rep=rep1&type=pdf
Microsoft, (2009). Project Natal Website, Xbox. http://www.xbox.com/en-US/live/projectnatal/
Nilsson, S. & Johansson, B.J.E. (2006). Acceptance of Augmented Reality Instructions in a Real
Work Setting. http://www.ida.liu.se/~susni/papers/chi952-Nilsson.pdf
QR Marker, 2009. http://blog.jactionscripters.com/2009/05/23/introduction-of-qr-code-readerlibrary/
Reiners, D., Stricker, S., Klinker G. & Mller S. (1998). Augmented Reality for Construction
Tasks: Doorlock Assembly. Proceedings of the international workshop on Augmented Reality:
placing artificial objects in real scenes, Natick, MA.
http://wwwbruegge.in.tum.de/publications/includes/pub/reiners1998iwar/reiners1998iwar.pdf
Schwald, B. & Laval B.d. (2003). An Augmented Reality System for Training and Assistance to
Maintenance in the Industrial Context.
http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.14.9192&rep=rep1&type=pdf
VUZIX, AR education group (2009). http://www.vuzix.com/AR_Site/default.asp
Yuan, M.L., Ong, S.K. & Nee, A.Y.C. (2005). Assembly Guidance in Augmented Reality
Environments Using a Virtual Interactive Tool. http://dspace.mit.edu/handle/1721.1/7442
Zhong, X., Liu, P., Georganas, N.D., & Boulanger, P. (2003). Designing a Vision-based
Collaborative Augmented Reality Application for Industrial Training.
http://www.discover.uottawa.ca/publications/files/ITjournal.pdf
Psychophysical Considerations in the Design of Human-Machine Interfaces for Tele-operator and
Virtual-Environment Systems
An Interactive Approach of Assembly Planning
2005. Boeing working on Augmented reality training
http://www.boeing.com/news/frontiers/archive/2006/october/i_ids03.pdf ,
http://www.boeing.com/defensespace/space/constellation/references/reports/Reduction_of_Space_Exploration_Risk_Use_of_IS
S_as_a_Testbed.pdf

37