Report

COMSATS Institute of Information Technology, Islamabad
MANIPULATION OF MICROSOFT POWERPOINT ®

SLIDES BY USING AUGMENTED REALITY
MARKER
By
Aniqa Hafeez
FA06-BCS-012
Mohsin Yazdanie
FA06--BCS-036
Supervisor: Dr. Muhammad Tahir

Co-Supervisor: Mr. Usama Khalid
Bachelor of Computer Science (2006-2011)
The candidate confirms that the work submitted is his own and appropriate credit has
been given where reference has been made to the work of others
Page ii
CERTIFICATE OF APPROVAL
It is to certify that the final year project of BS (CS) “Manipulation of Microsoft PowerPoint ®
Slides by using Augmented Reality Marker” was developed by “Aniqa Hafeez (CIIT/FA06-
BCS-012)” and “Mohsin Yazdanie (CIIT/FA06-BCS-036)” under the supervision of “Dr.
Muhammad Tahir” and “Mr. Usama Khalid” and that in their opinion; it is fully adequate, in
scope and quality for the degree of Bachelors of Science in Computer Sciences.
Supervisor Co-Supervisor
External Examiner
Head of Department
(Department of Computer Science)
Page iii
EXECUTIVE SUMMARY
Our project focuses on enhancing the user interaction with the computer by using the techniques
of Augmented Reality (AR) and Tangible User Interfaces (TUIs). It describes the manipulation
of Microsoft PowerPoint slides using AR Markers gestures (A marker is a simple paper with
specified pattern printed on it, having no physical connection with the computer). User performs
gestures by holding the AR Marker in hand to manipulate slides. The operations which this
application allows to perform are Go to Next slide, Go to Previous slide, Go to First slide, Go to
Last slide, Go to Specific slide (Next, Previous both), Highlight text (by crossing or pointing
techniques). When an AR Marker is brought in front of the camera, its pattern is detected.
Different movements of the marker constitute a gesture. These gestures are then recognized by
software module and the required operation is performed.
Page iv
ACKNOWLEDGMENT
We thank ALLAH for making us being able to seek the knowledge at its deepest extent and
utilize it, which helped us in making material contribution to the technology we worked on.
Additionally we would like to praise for the Holy Prophet (PBUH), who is forever a source of
guidance and knowledge.
We sincerely acknowledge the help and guidance of our supervisor, Dr. Muhammad Tahir, and
co-supervisor, Mr. Usama Khalid whose concern, involvement and encouragement gave us a
great support and motivation in doing this project.
Lastly we would like to thank Mr. Mustafa Madni and all the faculty members for providing help
and support throughout the period of this project.
Page v
DECLARATION
We hereby declare that this software, neither whole nor as a part has been copied out from any
source. It is further declared that we have developed this software and accompanied report
entirely on the basis of our personal efforts. If any part of this project is proved to be copied out
from any source or found to be reproduction of some other. We will stand by the consequences.
No Portion of the work presented has been submitted of any application for any other degree or
qualification of this or any other university or institute of learning.
Aniqa Hafeez Mohsin Yazdanie

Page vi
LIST OF ABBREVIATION
Acronyms Abbreviation/Definition
AR Augmented Reality
CIIT COMSATS Institute of Information Technology
CS Computer Science
CV Computer Vision
HCI Human Computer Interaction
SRS Software Requirements Specifications
TUI Tangible User Interface
TAR Tangible Augmented Reality
Page vii
REVISION HISTORY
Name Date Comments Version
SRS 14 April,2010 First SRS Document 0.1
Detailed design 14 May,2010 Detailed Design Document 0.5
Final draft 20 Dec,2010 Final Documentation 1.0

Page viii
Contents
1 INTRODUCTION……………………………………………………………………………..1
1.1 BRIEF ........................................................................................................................................... 2
1.2 RELEVANCE TO COURSE MODULE .............................................................................................. 3
1.3 PROJECT BACKGROUND............................................................................................................... 4
1.4 LITERATURE REVIEW................................................................................................................... 4
1.4.1 What is Augmented Reality (AR)? ......................................................................................... 4
1.4.2 WHAT IS TANGIBLE USER INTERFACE (TUI)? ........................................................................ 5
1.4.3 WHAT IS TANGIBLE AUGMENTED REALITY (TAR)?............................................................... 5
1.4.4 WHAT IS GESTURE-BASED INTERACTION? ............................................................................. 6
1.4.5 WHAT IS AN AR MARKER? ..................................................................................................... 6
1.4.6 WHAT IS ARTOOLKIT? ........................................................................................................... 7
1.4.7 PAPERS REVIEWED .................................................................................................................. 9
1.5 METHODOLOGY AND SOFTWARE LIFE CYCLE .......................................................................... 11
2 PROBLEM DEFINITION...………………………………………………………………....12
2.1 PROBLEM DEFINITION ................................................................................................................ 13
2.2 DELIVERABLES AND DEVELOPMENT REQUIREMENTS .............................................................. 14
2.2.1 DOCUMENTATION DELIVERABLES ........................................................................................ 14
2.2.2 APPLICATION REQUIREMENTS .............................................................................................. 14
2.2.3 COMMON COMPONENT DEVELOPMENT REQUIREMENTS ...................................................... 14
2.2.4 HARDWARE REQUIREMENTS ................................................................................................. 14
3 OUR WORK…………………………………………………………………………………..15
3.1 BRIEF OVERVIEW....................................................................................................................... 16
3.2 DIFFERENT TYPE OF MARKERS ................................................................................................. 17
3.2.1 ONE HANDED INTERACTION ................................................................................................ 17
3.2.2 TWO HANDED INTERACTION ................................................................................................ 20
3.3 CALCULATE VALUES OF DIFFERENT MARKER SIZES AND RANGES .......................................... 23
3.4 COMPARISON GRAPH OF DIFFERENT TYPE OF MARKERS SHOWING RANGE VALUES .............. 23
3.5 METHODS USED ......................................................................................................................... 24
3.6 SYSTEM WORKING..................................................................................................................... 29
3.7 LIMITATIONS.............................................................................................................................. 32
3.7.1 GHOST MARKERS ................................................................................................................. 32
3.8 GESTURES VOCABULARY........................................................................................................... 34
4 SPECIFIC REQUIREMENTS……………………………….…..………………………….38
4.1 FUNCTIONAL REQUIREMENTS ................................................................................................... 39
4.1.1 FACILITATE THE USER TO BROWSE THE SLIDES..................................................................... 39
4.1.2 MARKER DETECTION ............................................................................................................ 39
4.1.3 GESTURE RECOGNITION ........................................................................................................ 39
4.1.4 GO TO NEXT SLIDE ................................................................................................................. 39
4.1.5 GO TO PREVIOUS SLIDE .......................................................................................................... 39
4.1.6 GO TO FIRST SLIDE ................................................................................................................. 39
4.1.7 GO TO LAST SLIDE .................................................................................................................. 39
4.1.8 GO TO SPECIFIC SLIDE............................................................................................................ 39
4.1.9 HIGHLIGHT TEXT ................................................................................................................... 39
4.2 NON-FUNCTIONAL REQUIREMENTs ........................................................................................... 40
4.2.1 USABILITY ............................................................................................................................. 40
4.2.2 RELIABILITY .......................................................................................................................... 40
4.2.3 PERFORMANCE ...................................................................................................................... 40
4.2.4 SUPPORTABILITY ................................................................................................................... 40
4.2.5 DESIGN CONSTRAINTS .......................................................................................................... 40
4.2.6 USER DOCUMENTATION AND HELP SYSTEM REQUIREMENTS .............................................. 41
4.2.7 PURCHASED COMPONENTS ................................................................................................... 41
Page ix
4.2.8 INTERFACES ........................................................................................................................... 41

4.2.8.1 USER INTERFACE............................................................................................................ 41
4.2.8.2 HARDWARE INTERFACE ................................................................................................ 41
4.2.8.3 SOFTWARE INTERFACE .................................................................................................. 41
4.2.9 LICENSING REQUIREMENTS................................................................................................... 41
4.2.10 LEGAL, COPYRIGHT AND OTHER NOTICES ........................................................................... 41
5 SYSTEM DESIGN…………………………………………………………………………....42
5.1 USE CASE DIAGRAM .................................................................................................................. 43
5.1.1 CONNECT ............................................................................................................................... 44
5.1.2 MANIPULATE SLIDES............................................................................................................. 45
5.1.3 VIEW GESTURE HELP ............................................................................................................ 46
5.1.4 GO TO NEXT SLIDE ............................................................................................................... 47
5.1.5 GO TO PREVIOUS SLIDE ........................................................................................................ 48
5.1.6 GO TO FIRST SLIDE ............................................................................................................... 49
5.1.7 GO TO LAST SLIDE ................................................................................................................ 50
5.1.8 GO TO SPECIFIC SLIDE (NEXT) ............................................................................................ 51
5.1.9 GO TO SPECIFIC SLIDE (PREVIOUS) ...................................................................................... 52
5.1.10 HIGHLIGHT TEXT ON SLIDE .................................................................................................. 53
5.2 CLASS DIAGRAM ....................................................................................................................... 54
5.3 SEQUENCE DIAGRAM ................................................................................................................. 55
5.3.1 GO TO FIRST SLIDE ................................................................................................................ 55
5.3.2 GO TO LAST SLIDE ................................................................................................................ 55
5.3.3 GO TO NEXT SLIDE ................................................................................................................ 56
5.3.4 GO TO PREVIOUS SLIDE ......................................................................................................... 56
5.3.5 GO TO SPECIFIC SLIDE (NEXT) .............................................................................................. 57
5.3.6 GO TO SPECIFIC SLIDE (PREVIOUS) ....................................................................................... 57
5.3.7 HIGHLIGHT TEXT ON SLIDE ................................................................................................... 58
5.4 ACTIVITY DIAGRAM .................................................................................................................. 59
5.4.1 GO TO FIRST SLIDE ............................................................................................................... 59
5.4.2 GO TO LAST SLIDE ................................................................................................................ 59
5.4.3 GO TO NEXT SLIDE ............................................................................................................... 60
5.4.5 GO TO SPECIFIC SLIDE (NEXT).............................................................................................. 61
5.4.7 HIGHLIGHT TEXT ON SLIDE ................................................................................................... 62
5.5 STATE DIAGRAM........................................................................................................................ 63
5.5.1 GO TO FIRST SLIDE ............................................................................................................... 63
5.5.2 GO TO LAST SLIDE ................................................................................................................ 63
5.5.3 GO TO NEXT SLIDE ............................................................................................................... 64
5.5.5 GO TO SPECIFIC SLIDE (NEXT).............................................................................................. 65
5.5.7 HIGHLIGHT TEXT ON SLIDE .................................................................................................. 66
5.6 COMPONENT DIAGRAM ............................................................................................................. 67
5.7 DEPLOYMENT DIAGRAM ........................................................................................................... 67
6 IMPLEMTATION…………………………………………………………………………....68
6.1 USER MODULES AND CHARACTERISTICS .................................................................................. 69
6.1.1 CONNECT ............................................................................................................................... 69
6.1.3 VIEW GESTURE HELP ............................................................................................................. 69
6.2 USER INTERFACES ..................................................................................................................... 69
6.3 SOFTWARE QUALITY ATTRIBUTES ............................................................................................ 69
Page x
7 TESTING……………………………………………………………………………………...70
7.1 VERIFICATION ............................................................................................................................ 71
7.1.1 FUNCTIONAL TESTING........................................................................................................... 71
7.1.2 STATIC TESTING .................................................................................................................... 76
7.2 VALIDATION .............................................................................................................................. 76
7.3 USABILITY TESTING .................................................................................................................. 76
7.4 UNIT TESTING ............................................................................................................................ 76
7.5 INTEGRATION TESTING .............................................................................................................. 77
7.6 SYSTEM TESTING ....................................................................................................................... 77
8 USER GUIDE………………………………………………………………………………....78
8.1 GUI (GRAPHICAL USER INTERFACE)......................................................................................... 79
8.1.1 MAIN INTERFACE .................................................................................................................. 79
8.1.2 CONNECT ............................................................................................................................... 79
8.1.4 HELP ...................................................................................................................................... 82
9 CONCLUSION AND FUTURE WORK……………………………….……………..……..83
10 REFERENCES…………………………………..…………………………………………..85
1
CHAPTER
1
INTRODUCTION
2
1.1 BRIEF
This is a research based project which enhances the user interaction with the computer using
Augmented Reality (AR) and Tangible User Interfaces (TUIs) techniques.
AR has variety of definitions according to the research literature available, few are mentioned
below:-
 Augmented reality (AR) is a term for a live direct or indirect view of a physical real-
world environment whose elements are augmented by virtual computer-
generated sensory input such as sound or graphics. [1]
 Augmented reality (AR) is the registration of projected computer-generated images
over a user’s view of the physical world. With this extra information presented to the
user, the physical world can be enhanced or augmented beyond the user’s normal
experience. [2]
 Augmented reality (AR) is an engaging way of combining live video with computer-
generated data and visualizations. [3]
TUI is defined as:-
 Tangible user interface (TUI) is a user interface in which a person interacts with
digital information through the physical environment. [4]
 Tangible user interface (TUI) uses physical objects which are designed and
constructed at home or industry to control and manipulate digital information.[5]
In this project, user will be performing different gestures using AR Marker in front of the
camera to manipulate Microsoft PowerPoint slides.
Target Platform
System:- Windows OS Based
Operating System:- Windows XP or Later
Software:- Microsoft Office PowerPoint
Development Platform
Development Environment:- Visual Studio 2008

Toolkit:- ARToolKit 2.72
Language:- C++, C#
Framework:- .Net Framework 3.5
3
This document contains general overview of the Microsoft PowerPoint slides manipulation
using AR Marker including behavior diagrams, interaction diagrams, features, functionality
and technology used to create this application. The document is divided into different
sections helping readers to understand the system and its usage.
 Chapter 1 is the introduction, a brief overview of our system and it explains the
technology or concepts used in our project.
 Chapter 2 describes the problem definition, why we chose this project.
 Chapter 3 is the detailed explanation of our work, including everything that we
worked on for the project.
 Chapter 4 tells about the specific requirements of the system that includes functional
and non-functional requirements.
 Chapter 5 is the system detailed design.
 Chapter 6 contains the system implementation.
 Chapter 7 shows results of the system tests that we performed and evaluation.
 Chapter 8 is a user guide.
 Chapter 9 is about the conclusion and future work.
 Chapter 10 includes references.
1.2 RELEVANCE TO COURSE MODULE
The courses of BS (CS) degree, which helped us in the successful completion of this project,
include:
• Human Computer Interaction (HCI):- helped us to study people (users), different ways
of interaction with the computer and effective designing of user interfaces.
• Computer Vision (CV):- helped us to study different algorithms used for image
processing such as background subtraction, event detection, video tracking, object
recognition, learning, indexing, motion estimation, and image restoration etc.
• Software Engineering I and II: - provides us the guidelines, processes and techniques
that are necessary to select the methodology for the development of this project and
documentation.
• Algorithm and Data Structure: - helping in understanding different types of data
structures and algorithms as well as improving programming skills.
• Communication Skills: - was helpful in improving our presentation skills for the project.
4
1.3 PROJECT BACKGROUND
In this project we have used different AR Markers for performing gestures, whereas so far
AR Markers were used to display a 3D virtual model on it using different AR ToolKits. We
used AR Markers as a tool to interact with the computer helping user to manipulate slides
easily
1.4 LITERATURE REVIEW
Literature review helped us in knowing about Augmented Reality, AR markers, ARToolKit,

the maximum use of AR Marker, and the existing work related to our project. Our project is
about gesture-based interaction using AR markers. It involves marker detection and gesture
recognition. We used Augmented Reality (AR) in conjunction with Tangible User Interfaces
(TUIs) (which results in Tangible Augmented Reality (TAR)).
We searched several related topics and concepts about our project which are discussed below
in further sections. These explain: what is AR, what is an AR Marker, what is ARToolKit and
how they works, what is TAR, TUIs and gesture based interaction.
1.4.1 What is Augmented Reality (AR)?
Augmented Reality (AR) integrates computer generated world with real world. The objects
that come under real world can be any physical objects of any shape and size, for example a
cardboard, a plastic cover, a plane flat paper etc. The objects that come under computer
generated world are virtual objects which can be any computer-generated graphics. When
these virtual objects are plotted on the corresponding physical objects, by using software
toolkits, it creates an application related to Augmented Reality.
Kevin Bonsor says in his article that “Augmented reality is changing the way we view the
world -- or at least the way its users see the world. Picture yourself walking or driving down
the street. With augmented-reality displays, which will eventually look much like a normal
pair of glasses, informative graphics will appear in your field of view and audio will coincide
with whatever you see. These enhancements will be refreshed continually to reflect the
movements of your head.”[6]
5
Figure 1.1 Left: AR application using AR marker and human

Interaction [7], Right: Mobile AR application [8]
AR is becoming very helpful in making higher education more innovative and intuitive for
students by introducing new interactive teaching techniques.
1.4.2 WHAT IS TANGIBLE USER INTERFACE (TUI)?
Our project mainly deals with TUI. The TUI technology is used to manipulate digital
information by dealing with the physical objects. [9]
“A tangible user interface (TUI) is a type of user interface that uses physical form and
physical attributes as a way to help people understand how to access, grasp and manipulate
intangible digital information. Tangible interfaces provide physical handles to digital
information by building on people’s intuitive understanding and expectations of how physical
objects operate in the real world.” [10]
1.4.3 WHAT IS TANGIBLE AUGMENTED REALITY (TAR)?
Tangible Augmented Reality (TAR) is the combination of AR and TUI Systems. TAR
provides seamless display and interaction between real and digital worlds. Different TAR
applications have been used in various domains like education, entertainment etc. In the
education domain, several learning techniques have been proposed. One of them is the
6
Learning Cube (Terrenghi et al., 2006,[11] ) which is designed for helping students to answer
quizzes using a 3D cube. This cube allows different types of 3D movements. Answer to the
consequent question, either in form of text or images, is displayed on the screen of this cube
by performing these movements. This is interactive hence very helpful for students in
remembering [9].
1.4.4 WHAT IS GESTURE-BASED INTERACTION?
Gesture-based interaction is the user interaction with computer in which user has to perform
gestures to use applications on computer. These gestures can be performed by simply using
hands, fingers, face motions and positions or by using different tools such as Wiimote, AR
Marker etc.
It frees the hands of user from using keyboard or mouse and constantly sitting in front of the
computer.
1.4.5 WHAT IS AN AR MARKER?
A marker is a square shaped pattern to be printed or pasted on any flat surface; we have used
a plane paper as a flat surface. The idea was to use it as a physical tool to interact with the
computer. Such type of marker does not involve any circuitry or physical connectivity with
the computer. These markers can be detected by different AR Toolkits using web cameras
and computer vision algorithms.
These are some of the AR markers with different patterns:-
Kanji Sample 1 Sample 2 Hiro
Figure 1.2: ARToolKit Markers

7
We can make custom markers, keeping in mind following things [12]:
• Patterns should not be very complex.
• Black part of the pattern should be anti reflective material.
• White part should be clear enough or should include florescent material to increase
contrast in low light.
1.4.6 WHAT IS ARTOOLKIT?
ARToolKit is an open source C language library that helps programmers to create different
types of augmented reality applications. It is developed by Dr. Hirokazu Kato of Osake
University, Japan. It is designed for different type of platform e.g Linux, MacOS and
Windows OS. [13]
It uses computer vision algorithms to calculate the coordinates of marker and camera which
helps programmers to project any virtual object on an AR Marker relative to its position.
1.4.6.1 HOW ARTOOLKIT WORKS?
Figure 1.3: ARToolKit working Process stepwise

8
When ARToolKit starts and the camera is connected with the computer, it captures the real
world video and sends it to the ARToolKit API. Then it searches for the black square pattern
(that a marker consists) in the video. After that the 3D position and orientation of the marker
with respect to camera coordinates is calculated, the ARToolKit extracts the marker id from
the inside the black square pattern and compares it with the pre-defined patterns. If the
current pattern matches with any of the pre-defined patterns, the assigned 3D virtual object is
aligned on the marker. Then it is displayed on the monitor. [9]
Figure 1.4 Execution of ARToolKit
The marker’s tracking challenges in ARToolKit include:
• Camera and marker’s orientation

• Occlusion
• Unfocused camera, motion blur
• Jittering
• Ambient light
• Noise in camera image
• Speed of motion
9
ARToolKit is simple and fast in detecting single marker. ARToolKit is used for overlaying
virtual objects on markers. We have used it for capturing the gestures performed by AR
marker which has been less explored before [9].
1.4.7 PAPERS REVIEWED
We have reviewed research papers on human computer interaction, gestures, augmented

reality, different AR toolkits, markers, markers detection, and issues in detection before
implementing this system. The relevant information of these papers, regarding our project, is
described below.
 Measuring ARToolKit Accuracy in Long Distance Tracking Experiments [14]
This paper describes an experiment that was performed to compute the accuracy of
ARToolkit when the AR marker target distance is between 1m to 2.5m. It resulted that the
error in X and Y values increases as the distance increases and the accuracy of marker
detection changes as the angle to the target distance changes. This paper proposed that
using correction filter, the X and Y values can be corrected to 75% of the already
measured values. It says that for better understanding of the problems discussed here, it
would be good enough to repeat the experiment with some changes such as using
different camera models and lenses, performing same tests at different distances, testing
the proposed linear correction method(use filters), use different marker pattern.
As we are also using ARToolKit in our project so for finding the accuracy of Marker
detection at some changes in distance, this paper is helpful.
 Design of halftone-based AR markers under infrared detection [15]
A special invisible AR marker under IR has been proposed in this paper. Half toning is
used for printing the marker so that it becomes invisible for human eye. It says “Since the
human eye possesses the ability to integrate the neighboring halftone dots, the output
binary image then perceives like a con-tone image”[15]. Carbon black for blank ink (K)
with the combination of Cyan (C) Magenta (M) and Yellow (Y) inks under visible light is
proposed for the half toning so this way the hidden AR marker watermark can be
observed under IR light.
10
 Manipulating and annotating slides in a multi-display environment [16]
An interactive environment for giving presentations has been proposed with several
techniques to manipulate and annotate slides. This work has been done for keeping the
meeting or conference room more interactive by giving presentations in multiple wall
displays and allowing the audience or participants to annotate the slides during the
presentation with their “wireless pen notebook computers called NoteLook 3.0”.
It is related to our project as the applications that are developed here are for manipulating
slides. But here it has been performed using touch screen where as we are using AR
Markers for performing gestures to manipulate slides. This could also be helpful in
including this type of option in our project especially the option of displaying the slide in
multiple languages.
 Image Manipulation through Gestures [17]
A method is proposed in this paper for manipulating and editing the images on a filmstrip
i.e. performing AR marker gestures wearing the Fiducial marker on fingers and hands as
hand rings and bracelets. These AR gestures are tracked using ARToolkit. Moreover,
gesture combination have been used to perform certain tasks on an filmstrip images like
pick, drag, drop, delete, zoom, browse, stop and play. The gesture recognition is
performed by “topological analysis between visible markers centre vectors distances, mid
points and angles” [17]. We also used ARToolKit which is helpful in defining the AR
marker gestures for manipulation of slides, but for performing gestures we have used
different versions of AR marker.
 Method for Predicting Marker Tracking Error [18]
The quantification of errors during the real time marker tracking is discussed in this
paper. It says that most of the systems ignore the importance of quantifying the accuracy
of the pose that is being calculated. Further it says that “errors are strongly dependent on
camera parameters and scene geometry” [18]. Basically the corner and edges
identification tracking errors are considered and approaches are used to reduce these
errors so that they may not lead to larger errors. The “Numerical Sampling Technique
based upon SSSUT (Spherical Symmetric Unscented Transformation) and Scaled
Unscented Transformation based upon RPP (Planar Pose algorithm) optimizer” is used.
11
SSSUT predicts the translational errors. Based on this, we have used the translational
movements of the marker.
 Mouse cursor control using a head-mounted marker [19]
A method is proposed in this paper for mouse control using a simple AR head mounted
marker and a web cam. Two modes are compared 1) position control in which cursor
positions where the head is pointing and 2) velocity control in which cursor keeps on
moving with constant speed in the direction of head till user continues to the head
movement. This system is intended for disabled users who cannot use hands so only
head motion is required to move the mouse cursor. Our project also includes mouse
control using AR marker but through hand gestures, which does not bound the user to sit
at one place rite in front of the computer and then use the software because it is proposed
for giving presentations.
1.5 METHODOLOGY AND SOFTWARE LIFE CYCLE
The methodology we use for software life cycle is “Waterfall Method”. Firstly, we specified
the problem definition. Then the designing phase was done in detail. According to that design
we developed the system. Then the testing phase was done, the main and necessary types of
testing was done successfully.
12
CHAPTER
2
PROBLEM DEFINITION
13
2.1 PROBLEM DEFINITION
The basic tasks a presenter generally performs, during a presentation, include: change next
and previous slides, go to first, last and a specific slide and highlight text on a slide. Usually,
these tasks are performed by using input devices like keyboard, mouse or special remote
controls for presentations. There are certain issues involved in these input devices such as in
case of keyboard and mouse the presenter moves from its current position to computer and
then presses the required keys in order to perform the required tasks. This is not feasible for
presenters with walking disabilities. Similarly, for the tasks “go to a specific slide” and
“highlight text on a slide”, presenter first displays a context menu by pressing Shift+F10 from
the keyboard or by right clicking on the slide using mouse and then selects the required slide.
This results in unnecessary physical movements and also diverts presenter’s attention from
the audience. Furthermore this causes disturbance in the flow of the presentation. Although
certain operations like “change next and previous slides” are possible using special remote
controls, still, there are certain operations like highlight text and go to first, last and specific
slide which are restricted or sometimes even not possible with some of these devices.
Thus, in order to perform these tasks, it requires exploring new techniques of interaction. An
option to manipulate presentation slides is to use the techniques of Tangible User Interfaces
(TUIs) and Augmented Reality (AR). TUIs help in controlling computer‐generated
information by using physical objects while AR combines the real life objects with virtual
objects. A real‐life object can be any physical device such as a piece of cardboard, a wood
cube, a finger ribbon, etc. that could be used to manipulate digital contents. TUI and AR
(when combined called Tangible Augmented Reality (TAR)) techniques involve using
special type of physical objects called AR markers. An AR marker is of a flat shape and
contains a special pattern on it. This pattern is detected by a webcam and is processed by
using special toolkits such as ARToolKit.
These AR markers could be helpful in performing the above mentioned tasks because they
provide the continuity and flow of operations. This is achieved by using markers IDs,
translational and rotational movements. We use AR markers as new types of interaction
devices to manipulate Microsoft PowerPoint slides. Moreover, our system uses ARToolKit in
order to recognize different types of markers we designed. These AR markers as gesture
14
devices have been less explored for presentation scenarios. So we have defined AR Marker
gestures to manipulate Microsoft PowerPoint Slides.
2.2 DELIVERABLES AND DEVELOPMENT REQUIREMENTS
Following are the deliverables:
2.2.1 DOCUMENTATION DELIVERABLES

• User Manual.
• Final Report.
2.2.2 APPLICATION REQUIREMENTS
• .NET Framework.
• Office 2007.
• Windows operating system.
2.2.3 COMMON COMPONENT DEVELOPMENT REQUIREMENTS
• Smart Draw (for UML 2.0).
• Microsoft Office 2007 (for Report writing and presentation).
• Visual Studio 2008
• ARToolKit
2.2.4 HARDWARE REQUIREMENTS
• Desktop or Laptop System.
 Processor Intel Core2Duo 2.0 GHz
 RAM 2GB
 HDD minimum 40GB.
• Web camera (USB).
• AR Markers.
15
CHAPTER
3
O UR WORK
16
3.1 BRIEF OVERVIEW
Our project mainly focuses on the innovative human interaction with computer. We have
introduced AR Marker for interacting with computer to manipulate Microsoft PowerPoint
slides. This doesn’t require any additional physical connection/circuitry with computer and
it’s totally sensor-less. The main idea behind this is that user can move while giving
presentation without being bounded to use keyboard/mouse.
For markers detection’, we have used ARToolKit library. We further used “loadmutiple.c”
and modified it according to our project requirement. We used marker position values and
defined our own gesture vocabulary for each marker. Different patterns of AR Marker are
used to manipulate different PowerPoint functions by the help of PowerPoint office library.
We have also created new markers (customized markers) and trained them with ARToolKit
in order to be used in our project.
By using AR Markers, we have performed following operations:
Changing to:
 Next Slide
 Previous Slide
 First Slide
 Last Slide
 Go to Specific Slide (Next, Previous)
 Highlight Text (by crossing (Apitz et al. (2004)[20]) or pointing technique [21])
We have designed different types of markers with different sizes and patterns, their testing
and results are described in further chapters. The system specifications and web camera we
used are described below:-
System : HP Pavilion DV5 Core2 Duo P8400,
3gb Ram
Intel Graphics Card.
Camera : A4Tech Model PK-333[22]
OS : Windows 7 Ultimate 32 bit

17
3.2 DIFFERENT TYPE OF MARKERS
After literature review, we planned to design and test five different types of markers which lie
under the category of one handed and two handed interaction. These markers in our view help
users to interact with computer easily. Each customized marker is described below with their
construction and working details.
3.2.1 ONE HANDED INTERACTION
1 SIMPLE AR MARKER
 Construction
It is a simple marker consisting of a black square pattern and printed on a white paper.
Then it is properly pasted on some flat surface (e.g. card board / hard board) so that
there is no air or gap between paper and hard board. This results in smooth detection
of marker by camera.
 Working
The user simply holds this marker in his hand and performs gestures in front of the
camera. It’s simple to handle but limited operations can be performed. It works well
in normal light condition but doesn’t work in dark room conditions. But it sometimes
faces occlusion problem, because our fingers may overlap the pattern when we hold it
for directional (x,y axes) movement.
Figure 3.1 Simple Markers with ‘Hiro’ Pattern

18
2 CUBE MARKER
 Construction
A cube is a 3-dimensional object having six square faces. We used a wooden made
cube and pasted 5 different AR Markers on its 5 different sides. The remaining
unused side was fitted with a handle helping user to hold it in his hand to perform
different gestures. This handle is also useful in performing translational and rotational
gestures [9] as described in another section.
 Working
User holds the handle of the cube marker in his/her hand to perform different type of
operations. It also helps user to perform different operations using one hand. It can be
a rotational or translational gesture. It works well in normal light condition but poorly
work in dark/semi-dark room conditions. Because there are five patterns on this cube,
so sometimes it is difficult for a user to remember the working of each face of the
cube. For this, we have defined one side as the default side which remains in front of
the user and which also acts as a starting point for the gesture.
Figure 3.2 Cube Markers
As mentioned above in simple AR marker, when we hold the marker our fingers often
overlap the marker pattern which affects the detection. So to avoid this occlusion we
have made a cube marker which has a handle to hold it in hand. Sometimes it results
in detection of more than one marker either from top or sides of the room. Sometimes
19
it considers surrounding objects as a marker and displays virtual augmented image.

These are called ghost markers which are described further in this chapter.
3 ILLUMINATED MARKER
 Construction
We used simple AR marker printed on a paper and pasted on a transparent CD cover.

After that we attached custom made LED circuit (white LEDs) at the back of that CD
cover so that it lightens up the marker (shown in the Figure 3.3). The LED circuit is
powered with a battery that turns on the LEDs. No extra cable or connectivity is
required to establish the connection between marker and the system.
 Working
This marker has the dual functionality. In dark room conditions it works well because
the marker’s background is illuminated by LED board. This lightens up the pattern
and eases the marker detection. While in normal light conditions it has no effect on its
detection because it works same as simple AR Marker.
It faces detection issues when the illumination from back of marker is uneven. Also,
the detection problem occurs when the illumination is above normal because it
brightens the surface too much and marker pattern becomes invisible.
Figure 3.3 ‘Hiro’ Pattern illuminated Marker

20
4 RETRO-REFLECTIVE MARKER
 Construction
We printed an AR Marker on a special type of reflective sheet [21] and pasted it on a

flat surface card board similar to simple AR Marker.
 Working
The user will hold this marker in his hand and will perform gestures. It has an average
performance in detection as compared to simple AR Marker because reflected light
causes pattern to get undetected by the camera.
Figure 3.4 ‘Hiro’ Pattern Retro-reflective Marker
3.2.2 TWO HANDED INTERACTION
1. FINGER MARKERS
 Construction
We designed small size of markers which can be easily used on fingers. We printed
different AR Markers in small equal sizes of 3.5cm and pasted on different small flat
card board attached with stretchable plastic ring.
21
 Working
These markers are easily wearable which a user can wear on his fingers and can easily
perform multiple operations.
Figure 3.5 Finger Markers

22
 COMBINATION OF FINGERS MARKERS
For two handed interaction multiple markers can be used at a time. For this purpose
we defined combinations of patterns to perform a specific function. Here in finger
markers we made a custom marker named ‘Neutral’ (wearing in last finger of both
hands in above figure) and made it recognized with ARToolKit.
Neutral
Figure 3.6 Custom Marker used to go to specific slide function.
The combination of ‘Neutral’ and ‘Hiro’ pattern is used for performing the function
‘go to specific slide (next)’.
The combination of ‘Neutral’ and “Kanji” pattern is used for performing the function
‘go to specific slide (previous)’.
Other than these two combinations, if any other combination is used it will not
perform any new function as it is not defined in our project so far. Moreover, using
above mentioned patterns of finger markers, for making a new combination, will not
perform any new single function. But it will first perform the function against the
pattern that is detected first and then the one that is detected second and so on. These
unused possible combinations could be used in future studies.
23
3.3 CALCULATED VALUES OF DIFFERENT MARKER SIZES AND RANGES
Maximum Range
Size of (inches)
Marker 𝟏 𝟐 𝟑 𝟒 𝟓 𝟔 𝟕 𝟖 𝟗 𝟏𝟎 Average
3.5cm 38 39 37 36 38 39 40 41 39 39 39
5.5cm 65 67 69 66 70 69 68 70 70 69 69
7.5cm 100 101 102 100 99 100 100 102 103 98 101
10cm 122 124 123 128 130 132 131 131 129 130 128
Different sizes of marker were used to calculate their maximum detection range. We used
default “Hiro” pattern printed on a paper in four different sizes (3.5cm, 5.5cm, 7.5cm, and
10cm). Their maximum detection range was calculated 10 times and their average was taken.
The test was done in a room with normal light condition and a4tech camera (Model: pk-
333mb [22]) was used. The result is shown in the graph by categorizing them in poor and
good detection.
3.4 COMPARISON GRAPH OF DIFFERENT TYPES OF MARKERS SHOWING

RANGE VALUES
140
120
100
80
Range
(in inches)
60 Poor Detection
Good Detection
40
20
0
3.5 5.5 7.5 10
Marker Pattern (in cm)
24
3.5 METHODS USED
We proposed different methods and used them separately and combined for manipulating
PowerPoint slides. Those two methods are described below:-
1) By using Marker IDs
In Marker IDs, each marker has been assigned a unique number in order to distinguish it
from other markers. The IDs and markers used here are the default IDs and markers of
ARToolKit. We also defined our own IDs and markers which are used as finger markers.
The patterns we used along with their IDs are given below:
Pattern name: Sample 1 Pattern name: Sample 2

ID: 2 ID: 3
Figure 3.7 Default Markers by ARToolKit. Used it for “go to specific slide” function.
The functions Go to Specific Slide (Next) and Go to Specific Slide (Previous) is

performed with the Marker IDs due to the following reasons. Firstly it frees the
presenter/user from displaying a context menu first and then to select the required slide.
In our project, by using only IDs of markers we directly performed the required function.
This is achieved when the marker is brought in front of the camera. It is beneficial
because it 1) saves the presenter from wasting time in unnecessary physical movements,
2) does not divert presenter’s attention from the audience, and 3) keeps a good flow of
the presentation. Moreover, it is easy to use because user doesn’t have to learn any
gesture for this operation.
When the application is running, user just brings a specific pattern in front of the camera
and action against that pattern is performed. The application interprets the ID and
performs the coded function against that ID. More than one function cannot be performed
by using only one ID as one ID is used for only one function (i.e. one ID is directly
proportional to one function). For having more than one choice of functions with one
25
marker we used gestures, which are explained in further sections. Here we have a general
diagram giving an overview of manipulation of slides using simple marker id. Suppose
for go to specific slide using sample 1 marker.
Figure 3.8 Manipulation of Slides using Marker ID
2) By using ID plus Marker Gesture
Marker ID is used along with the marker gesture to perform the function of highlighting
text. Working of ID is described above; gesture is added for performing this function
because for drawing on the text or slide continuous movement of marker is required. The
pattern used for this function is shown below along with its ID:
Pattern name: Kanji
ID: 1
Figure 3.9 Default Marker by ARToolKit. Used for highlighting text.

26
When the application is running, User brings the marker in front of the camera its pattern
and ID is checked and matched with the defined patterns and IDs. If the pattern is ‘Kanji’
the application interprets it’s ID and ‘highlight text’ function is selected. A pen pointer is
selected and as user keeps on moving the marker. The corresponding line keeps on
drawing on the text or slide. This function has some limitations which are mentioned in
‘Limitations’ section below.
Figure 3.10 Highlight Text on Slide using Marker ID and gesture
3) By using simple Marker Gestures
User performs the defined gestures for these operations: Go to Next Slide, Go to
Previous Slide, Go to First Slide, and Go to Last Slide by using e.g. “Hiro” pattern.
Pattern name: Hiro
ID: 0
Figure 3.11 ‘Hiro’ pattern used for gesturing

27
Here we have a step-wise diagram showing to manipulate slides by performing gestures.
Figure 3.12 Slides manipulation using Marker gestures
We used two types of gestures: Translational and Rotational.
i. In translational gestures we explored 2D (x, y) values of each marker (i.e.

horizontal and vertical gestures).
Figure 3.13 Translational movements

28
The marker and the hand containing the marker both moves horizontally towards
+ x and – x axis to perform horizontal gestures for Next slide and Previous slide,
and towards + y and – y axis to perform vertical gestures for First slide and last
slide respectively (as shown in Figure 3.8).
ii. Rotational gestures include Roll, Pitch and Yaw values; these are the
movements of marker which are measured in angles. Pitch is up and down, Yaw
is left and right and roll is rotation (clockwise/anti-clockwise). We emphasized on
the roll values of each marker because it helped us in performing another
operation: “Fast Forward or Backward movement of slides”. In this case the hand
does not move, only cube is rotated clockwise and anti-clockwise using its
handle. The clockwise rotation moves the slides forward and anti-clockwise
moves the slides backward continuously. This all depends upon the rotation angle
generated when user moves the handle in corresponding direction.
Figure 3.14 Upper Left: The possible rotational orientations with cube
marker [6] Upper Right: rotational roll movements. Lower Left: Clockwise
Rotational gesture. Lower Right: Anticlockwise Rotational gesture.
29
Dead Zone: There is always a dead zone defined in rotational gestures at which
no operations performed. Basically, 0 degree angle is the phase when the marker
facing the camera is completely balanced from all sides and is not tilted towards
any direction. This dead zone is made so that it gives a starting point to any of the
two operations and when the operation is performed due to the corresponding
angle generated, the marker has to come back to 0 degree (dead zone) to stop on
the particular slide.
In our system, we defined it between -10 to 0 or 0 to 10 degree angle on

respective right or left side because hand movement is in 3D (air) is not precise.
So if it is 0 degree it is difficult for the hand to remain balance at this point. So
when AR Marker is in between the defined angle no functionality is performed.
3.6 SYSTEM WORKING
If the user performs a gesture in front of a web camera using AR Marker then that gesture is
detected and matched with pre-defined gesture vocabulary. If the gesture is recognized and
matched then the defined operation against that gesture is performed. The operations can be
any of these: go to next slide, previous slide, go to first slide, go to last slide, go to specific
slide (Next, Previous), and highlight text on slide. These all operations are performed using
cube marker. The user has to keep the cube marker holding its handle in hand with the
required pattern facing the camera.
How to perform the required function is described below:
 Go To Next Slide
User brings the marker in front of the camera and performs straight line translational
gesture towards +X axis to go to next slide. We used ‘Hiro’ pattern marker for this
function.
 Go To Previous Slide
gesture towards -X axis to go to previous slide. We used ‘Hiro’ pattern marker for this
function.
30
 Go To First Slide
gesture towards +Y axis to go to first slide. The pattern we used for this function is
‘Hiro’.
 Go To Last Slide
gesture towards -Y axis to go to last slide. The pattern we used for this function is ‘Hiro’.
 Go To Specific Slide (Next)
User brings the ‘sample 1’ pattern marker in front of the camera and slides start shooting
in forward order to go to specific slide (Next).when user has to stop at specific slide
he/she takes away the pattern from camera view.
 Go To Specific Slide (Previous)
User brings the ‘sample 2’ pattern marker in front of the camera and slides start shooting
in backward order to go to specific slide (Previous).when user has to stop at specific slide
he/she takes away the pattern from camera view.
 Highlight Text
User brings the ‘Kanji’ pattern marker in front of the camera a pen pointer is selected and
as user keeps on moving the marker the line keeps on drawing on the text or slide for
highlight text. This is done by crossing technique [20].
31
Figure 3.15 User manipulating the slides using AR Marker

32
3.7 LIMITATIONS
There are few limitations regarding the use of our application. This includes detetction,
gesturing, environment and software limitations.The detection of marker is limited as: 1)
there is a range limitation for each marker depending upon its size, 2) if marker is out of
camera view then there is no detection and 3) in cube marker, while performing gestures the
side markers are at times detected and their functionality is performed unwantedly.
The limitations while performing gestures are that gestures are not much smooth at higher
distances. The environmental limitations include the limited light conditions and also the
background or the surface behind or beside the user which is captured by the camera. This
environment should be plane and not filled with objects or even persons as ARToolKit
detects the ghost markers sometimes. Furthermore, the software limitations are that only one
slide show at a time can be manipulated.
3.7.1 GHOST MARKERS
Sometimes the ARToolKit results in the detection of false markers either by mistakenly
capturing the objects that are placed near camera. Furthermore, it also considers the objects
from the surroundings such as wall corner, roof corner, human face features (eye, lips),
clothes etc as a marker and displays virtual image. These are called ghost markers.
In our project, while performing any operation with the respective marker, if ghost markers
are detected it affects the working of operation. Because when the ghost marker is detected
the ARToolKit considers it as some default marker and interprets its ID. The function defined
against that ID is performed redundantly while at the same time user tries performing his
required function holding the respective marker in his hand. As a result either one of the
functions is performed or both are performed on the first come first serve type detection
behavior.
These are some of the ghost markers captured in our project lab while using ARToolKit.
33
34
Figure 3.16 Ghost markers captured in lab
3.8 GESTURES VOCABULARY
1. Translational gestures
Translational gestures are used for changing to Next slide, Previous slide, First slide and
Last slide by performing horizontal and vertical gestures.
• Horizontally (Left, Right)
The marker and the hand containing the marker both moves horizontally towards
+ X and – X axis to perform horizontal gestures.
35
 Towards right, straight lines translational gesture is used for go to Next

slide function using ‘Hiro’ pattern marker.
Figure 3.17 Translational Gesture for Next Slide
 Towards Left, straight lines translational gesture is used for go to Previous

Figure 3.18 Translational Gesture for Previous Slide
• Vertically (Top, Bottom)
The marker and the hand containing the marker both moves vertically towards +
Y and – Y axis to perform vertical gestures.
 Towards top, straight lines translational gesture is used for go to First slide
function using ‘Hiro’ pattern marker.
Figure 3.19 Translational Gesture for First Slide

36
 Towards bottom, straight lines translational gesture is used for go to Last

Figure 3.20 Translational Gesture for Last Slide
2. Rotational gestures
Rotational values are the roll values of each marker. We used them for performing “Fast
Forward or Backward movement of slides”. In this case the hand does not move, only cube
is rotated clockwise and anti-clockwise using its handle.
1. Clockwise
The clockwise rotation moves the slides forward one by one or continuously
depending upon the degree of angle.
Figure 3.21 Rotational Gesture (Clockwise) for Previous Slide

37
2. Anti-clockwise
The anti-clockwise rotation moves the slides backward one by one or

continuously depending upon the degree of angle.
Figure 3.22 Rotational Gesture (Anti-Clockwise) for Next Slide

38
CHAPTER
4
SPECIFIC REQUIREMENTS
39
4.1 FUNCTIONAL REQUIREMENTS
The software should provide the following functionalities:
4.1.1 FACILITATE THE USER TO BROWSE THE SLIDES
The software should allow the user to browse for the required PPS (PowerPoint
Slideshow) file from anywhere it is placed on the computer.
4.1.2 MARKER DETECTION
The software must detect the Marker properly to continue performing any task.
4.1.3 GESTURE RECOGNITION
If performed gesture is correct it must be recognized by the application.
4.1.4 GO TO NEXT SLIDE
It must show the Next slide successfully.
4.1.5 GO TO PREVIOUS SLIDE
It must show the Previous slide successfully.
4.1.6 GO TO FIRST SLIDE
It must go to the First slide of presentation successfully.
4.1.7 GO TO LAST SLIDE
It must go to the Last slide of presentation successfully.
4.1.8 GO TO SPECIFIC SLIDE
It must go to the Specific slide as required by the user successfully.
4.1.9 HIGHLIGHT TEXT
It must allow the user to highlight text through crossing technique.

40
4.2 Non-Functional Requirements
4.2.1 USABILITY
The application and the interface should be very easy to use for every type of user. Simple
gestures are defined to manipulate slides helping the user to learn those gestures in less time.
4.2.2 RELIABILITY
System will be tested for reliability and to handle all possible errors. And defined Gestures
will not be very specific it will work under flexible Gesture movements with AR Marker.
4.2.3 PERFORMANCE
System should be loaded as quickly as possible and consuming less memory resources with
keeping consistent computer performance although its performance is totally dependent upon
the hardware technology.
4.2.4 SUPPORTABILITY
The system will be updated with new gesture detection techniques in efficient manipulation
of slides.
4.2.5 DESIGN CONSTRAINTS
 Time is the main constraint in detection and recognition of the gesture. So in less time
gesture should be recognized.
 Camera Range is also a constraint in detection of gestures. So a good camera within

range of AR marker should be used.
 Light condition is the biggest constraint in detection of gesture.
 Hardware should be fast enough to support intense image processing.

41
4.2.6 USER DOCUMENTATION AND HELP SYSTEM REQUIREMENTS
User documentation and help system should be available.
4.2.7 PURCHASED COMPONENTS
 Camera
4.2.8 INTERFACES
4.2.8.1 USER INTERFACE
The user interface should be simple and easy to use. The different options displayed will be
Connect, Manipulate slides, Help and Exit.
4.2.8.2 HARDWARE INTERFACE
Camera should be connected with the system so that the gesture performed by the user can be
recognized.
4.2.8.3 SOFTWARE INTERFACE
The software is developed using C# language based on Microsoft Visual Studio 2008.
4.2.9 LICENSING REQUIREMENTS
The system has no licensing requirements.
4.2.10 LEGAL, COPYRIGHT AND OTHER NOTICES
The software or any of its components cannot be used for any commercial purpose without
any agreement with the owner of the software.
42
CHAPTER
5
SYSTEM DESIGN
43
5.1 USE CASE DIAGRAM

44
5.1.1 CONNECT
Use Case Name: Connect.

Actors: User, System
Purpose: To connect with ARToolKit and camera.
Overview: The user will choose the connect option if he wants to manipulate
slides, ARToolKit and camera will be activated.
Pre-conditions: Camera should be attached.
Post-conditions: The user successfully connects with ARToolKit.
Actor’s Actions: System’s Response:
• User can choose camera resolution. • The system will display camera
resolution window.
• The system will display the camera
view window.
Alternative sequences:
• Camera Resolution window is not • System didn’t find any camera.
displayed to user.
• User has selected the resolution but
camera view window didn’t appear.
45
5.1.2 MANIPULATE SLIDES
Use Case Name: Manipulate Slides

Actors: User, System
Purpose: To browse the PowerPoint slides
Overview: The user will first browse the desired PowerPoint Slide Show file and
will run it.
Pre-conditions: System should be connected with camera.
Post-conditions: PowerPoint slides will be launched.
• User will browse the PowerPoint slides. • System will run the specified PowerPoint
slides.
• User is not selecting the supported • The system is unable to open slides.
format.
46
5.1.3 VIEW GESTURE HELP
Use Case Name: View Gesture Help.

Actors: User
Purpose: To get gesture help.
Overview: Help to learn gestures and information about different type of markers.
Pre-conditions: System main interface should be displayed.
Post-conditions: User will be successfully be able to view help.
• The user has viewed Gesture help. • The system will show gesture help.
• The gesture help file is not launched.
47
Use Case Name: Go To Next Slide.

Actors: System
Purpose: To move to next slide.
Overview: User will perform a gesture and that gesture will be matched and
according to that slide will be manipulated to next slide.
Pre-conditions: PowerPoint slides should be running.
Post-conditions: Successfully moved to Next Slide.
• User will perform gesture in front of the • The slide will move to next slide.
camera.
• AR Marker is not in camera range. • AR Marker is not being detected.
• Gesture is not completed. • Slide will not change.
48
Use Case Name: Go To Previous Slide.

Actors: System
Purpose: Move to Previous slide.
according to that slide will be manipulated to Previous slide.
Post-conditions: Successfully moved to Previous Slide.
• User will perform gesture in front of the • The slide will move to previous slide.
camera.
49
Use Case Name: Go To First Slide.

Actors: System
Purpose: Move to First slide.
according to that slide will be manipulated to First slide.
Post-conditions: Successfully moved to First Slide.
• User will perform gesture in front of the • The slide will move to First slide.
camera.
50
Use Case Name: Go To Last Slide.

Actors: System
Purpose: Move to Last slide.
according to that slide will be manipulated to Last slide.
Post-conditions: Successfully moved to Last Slide.
• User will perform gesture in front of the • The slide will move to Last slide.
camera.
51
5.1.8 GO TO SPECIFIC SLIDE (NEXT)
Use Case Name: Go to Specific Slide (Next)

Actors: System
Purpose: Move to Specific Slide in forward direction.
Overview: User will show a AR Marker in front of the camera and a PowerPoint
Next shoot operation will be performed according to that slide will be
manipulated to Specific slide.
Post-conditions: Successfully moved to Specific Slide.
• User will show the AR Marker in front of • The slide will move to Specific slide.
the camera.
• Slide will not change.
52
5.1.9 GO TO SPECIFIC SLIDE (PREVIOUS)
Use Case Name: Go to Specific Slide (Previous)

Actors: System
Purpose: Move to Specific Slide in backward direction.
Overview: User will show a AR Marker in front of the camera and a PowerPoint
Previous shoot operation will be performed according to that slide will be
manipulated to Specific slide.
Post-conditions: Successfully moved to Specific Slide.
• User will show the AR Marker in front of • The slide will move to Specific slide.
the camera.
53
5.1.10 HIGHLIGHT TEXT ON SLIDE
Use Case Name: Highlight Text on Slide.

Actors: System
Purpose: To highlight text using crossing technique
Overview: User will show the defined AR Marker for highlighting in front of
camera to active highlighter and then user will perform the gesture to
highlight desired text.
Post-conditions: Successfully Text highlighted
• User will show the AR Marker in front of • The text on the slide will be highlighted.
the camera.
• User will perform the gesture to highlight
54
5.2 CLASS DIAGRAM

55
5.3 SEQUENCE DIAGRAM

56

57

58

59
5.4 ACTIVITY DIAGRAM

60

61

62

63
5.5 STATE DIAGRAM

64

65

66

67
5.6 COMPONENT DIAGRAM
5.7 DEPLOYMENT DIAGRAM

68
CHAPTER
6
IMPLEMENTATION
69
6.1 USER MODULES AND CHARACTERISTICS
6.1.1 CONNECT
The user connects with the application in order to start marker detection. This functionality is
used so that user doesn’t have to activate the camera and ARToolKit separately. It will save
time of user.
The user can browse for the required .pps or .ppsx file and start manipulating the slides.
User can view the gestures and application help so that before using the application user is
well known of the gestures to perform and the way application is used for manipulating
slides.
6.2 USER INTERFACES
1. The Graphical User Interface (GUI) is provided for user to interact with the application
easily (easy-to-use).
2. The GUI is simple and aesthetically appealing so that user’s attention stays on the use
of application, not only on the interface (as it will save his/her time).
3. Minimum options are provided on the GUI for user to use the application so that
he/she doesn’t waste time in just interacting with the interface and choosing the
options.
6.3 SOFTWARE QUALITY ATTRIBUTES
1. The application will fulfill the key points of Human Computer Interaction, which
include simple, interactive, user friendly and self explanatory interface for the use o f
the application. Guidance is available for novice users.
2. The user will be provided with the minimal number of options so that user does not
need to find or recall so many things.
3. The help will also be provided so that the user can easily understand the application
and how to perform gestures.
4. The functionality of each module will be easily understandable by looking at the GUI.
70
CHAPTER
7
TESTING AND EVALUATION
71
Following testing was performed during the testing phase of the software life cycle.
7.1 VERIFICATION
Verification is a process of evaluating a system according to your system specification and

this process is also done in order to check that whether the software is right or not.
7.1.1 FUNCTIONAL TESTING
There are some test cases to test several functions of the system independently.
• TEST CASE 1:
This test case has been tested and correctness of this test case has been proved according to
the inputs and outputs are given below:
Test Cases Id T1
Objectives To check the process of shifting to Next Slide.
Environment Windows
Process Move to the Next Slide if the gesture is performed properly and
correctly using Hiro pattern of marker.
Result User has moved on to the Next Slide successfully.
• TEST CASE 2:
Test Cases Id T2
Objectives To check the process of shifting to Previous Slide.
Environment Windows
Process Move to the Previous Slide if the gesture is performed properly

and correctly using Hiro pattern of marker.
Result User has moved on to the Previous Slide successfully.
72
• TEST CASE 3:
Test Cases Id T3
Objectives To check the process of shifting to First Slide.
Environment Windows
Process Move to the First Slide if the gesture is performed properly and
Result User has moved on to the First Slide successfully.
• TEST CASE 4:
Test Cases Id T4
Objectives To check the process of shifting to Last Slide.
Environment Windows
Process Move to the Last Slide if the gesture is performed properly and
Result User has moved on to the Last Slide successfully.
73
• TEST CASE 5:
Test Cases Id T5
Objectives To check the process of highlighting text.
Environment Windows
Process When the user brings the Kanji pattern of marker in front of
camera, a pointer is selected for using crossing technique to
highlight text.
Result Screen is updated with the highlighted text.
• TEST CASE6:
Test Cases Id T6
Objectives To check if user can go to the specific slide (Next)
Environment Windows
Process When the user brings sample 1 pattern in front of the camera
and it is detected, it will start moving to the next slide until the
user stops at the specific slide.
Result The specific slide required by the user appeared successfully.
74
• TEST CASE 7:
Test Cases Id T7
Objectives To check if user can go to the specific slide (Previous)
Environment Windows
Process When the user brings sample 1 pattern in front of the camera
and it is detected, it will start going back to the Previous slides
until the user stops at the specific slide.
Result The specific slide required by the user appeared successfully
• TEST CASE 8:
Test Cases Id T8
Objectives To connect with ARToolKit.
Environment Windows
Process When the user clicks on connect, camera and ARToolKit connects
with our application.
Result Connection is established successfully, and marker detection
module started.
75
• TEST CASE 9:
Test Cases Id T9
Objectives To check when two or more slide shows are open.
Environment Windows
Process Two slide shows are open, one through our application and one just
simply without using our application. The user ends the slide
manipulation of the first slide show that was opened using our
application; The functionality of slide manipulation should not shift
to the second slide show that was also active.
Result Results are successful. The functionality of manipulation does not
shift to the second slide show that was not opened using our
application.
76
7.1.2 STATIC TESTING
The syntax of the system is properly tested and verified during the compilation of the system.
90% to 95% of errors are removed during the compilation and the rest of the logical errors
are removed during the final results evaluations.
7.2 VALIDATION
To observe the system behavior under the blunder/slip/fault conditions, different false
conditions have been applied to the system and the response of system is checked.
• Using some other pattern instead of Hiro for gesturing: The system will first check the
pattern of marker facing the camera, if it’s not Hiro then it will have no affect on the
application by performing gestures.
• Using some other pattern instead of Kanji for highlighting: If the user selects any other
pattern for highlighting, it does not start highlighting.
• Using marker that is partly in front of the camera: If the user brings the marker partly in
front of the camera and its square pattern is completely visible to camera, it does not have
any effect on application and marker is not detected.
7.3 USABILITY TESTING
The system is exposed to the real users who were new to this application, they used it and
then they give us a feedback about the project advantages and disadvantages. They also told
us the additional features that can be added.
7.4 UNIT TESTING
Each module is tested independently as soon as each module is implemented. Each module is
tested right after its implementation so that it can be error free and provide accurate results to
next module. The results of unit testing are also described above in functional testing phase.
77
7.5 INTEGRATION TESTING
After the unit testing all the modules are merged to have main functionality of project. When
the system is integrated it is tested again thoroughly and we remove all the bugs that appear
after integration of the modules.
7.6 SYSTEM TESTING
The system is checked or tested with respect to its environment that in which it will run or is
mentioned in the document. It was properly tested in the Microsoft Windows XP/ Microsoft
Windows Vista/7 32bit and ensured that it will give the required output.
78
CHAPTER
8
USER GUIDE
79
8.1 GUI (GRAPHICAL USER INTERFACE)
8.1.1 MAIN INTERFACE
8.1.2 CONNECT
80
User will not be able to click on the “Manipulate Slides” option until he/she first click on
connect, a camera window will open asking for resolution. Choose 320 X 240 as default
resolution.
81
Press OK or apply. A camera live video will open. Now user will be able to manipulate
slides.

82
Clicking on manipulate slides allows the user to browse the required .pps or .ppsx files.

83
CHAPTER
9
CONCLUSION AND FUTURE
WORK
84
The main objective of this project was to manipulate the Microsoft PowerPoint slides by
using augmented reality (AR) marker. There are a lot of different ways to manipulate slides
using keyboard, Mouse, Wii Remote, Laser etc. We used Augmented Reality Marker to
manipulate slides because it requires no physical connectivity with the PC and provide
intuitive interaction. We have used different ways to manipulate slides using multiple
markers e.g. finger markers, cube marker etc.
In future work, different ways can be introduced to manipulate slides using new AR toolkits.
We can define some more gestures to be performed using AR marker. Designing and testing
of different AR markers such as spherical markers, markers with LEDs, markers with laser
pointers etc. Making a virtual graphical environment of fairyland/Disneyland for kids’
entertainment and on educational perspectives such as designing AR books etc. is relevant
future work. We also want to extend our work as studying and removing ghost markers that
appear in background during marker detection.
85
CHAPTER
10
REFERENCES
86
1 – Augmented Reality from http://en.wikipedia.org/wiki/Augmented_reality
2 – Tinmith AR System, Augmented Reality and Wearable Computer Research

http://www.tinmith.net/
3 - http://econsultancy.com/uk/blog/4288-10-mindblowing-augmented-reality-apps-and-
videos
4 – Tangible User Interface from http://en.wikipedia.org/wiki/Tangible_User_Interface
One of the pioneers in tangible user interfaces is Hiroshi Ishii, a professor in

the MIT Media Laboratory who heads the Tangible Media Group. His particular vision
for tangible UIs, calledTangible Bits, is to give physical form to digital information,
making bits directly manipulable and perceptible. Tangible bits pursues seamless
coupling between these two very different worlds of bits and atoms.
5 - Tahir, M., Tangible and Tactile Interaction Techniques for Multimedia Systems. PhD
Thesis, Telecom ParisTech, Paris, France pg 29
6 - http://www.howstuffworks.com/augmented-reality.htm
7 – http://www.crunchgear.com/2008/07/20/lonely-try-fondling-augmented-reality-cyber-
maid-alice/
8 – http://www.techeblog.com/index.php/tech-gadget/augmented-reality
9 - Tahir, M., Tangible and Tactile Interaction Techniques for Multimedia Systems. PhD
Thesis, Telecom ParisTech, Paris, France
10 - http://dkds.ciid.dk/py/tangible-user-interface/overview/
11 - Terrenghi, L., Kranz, M., Holleis, P., and Schmidt, A. 2006. A cube to learn: a
tangible user interface for the design of a learning appliance. Personal Ubiquitous
Comput. 10, 2-3(Jan. 2006), 153-158.
12 – IEEE Virtual Reality 2007 March 10-14, 2007
13 – ARToolkit http://www.hitl.washington.edu/artoolkit/
ARToolKit was originally developed by Dr. Hirokazu Kato, and its ongoing development
is being supported by the Human Interface Technology Laboratory (HIT Lab) at the
University of Washington, HIT Lab NZ at the University of Canterbury, New Zealand,
and ARToolworks, Inc, Seattle.
87
14 - 10 - Pierre Malbezin, Wayne Piekarski and Bruce H. Thomas, Measuring

ARToolKit Accuracy in Long Distance Tracking Experiments, IEEE, 1st international
Augmented reality Toolkit Workshop, September 29, 2002 – Darmstadt, Germany .
15 - Hsi-Chun Wang, Wen-Hsin Liu, Chia-Long Chang, Yi-Hui Chen, Design of

halftone-based AR markers under infrared detection, 2008 International Conference on
Computer Science and Software Engineering IEEE, Taipei, Taiwan.
16 - Patrick Chiu and Qiong Liu and John Boreczky and Jonathan Foote and Tohru
Fuse and Don Kimber and Surapong Lertsithichai and Chunyuan Liao, Manipulating
and annotating slides in a multi-display environment, Proc. of INTERACT ’03, 2003, pg
583—590.
17 - José Miguel Salles Dias, Nuno Barata, Pedro Nande, Pedro Santos, André Correia,
Image Manipulation through Gestures, ADETTI/ISCTE, Associação para o
Desenvolvimento das Telecomunicações e Técnicas, de Informática, Edifício ISCTE,
1600-082 Lisboa, Portugal.
18 - Freeman, R. M., Julier, S. J., and Steed, A. J. 2007. A Method for Predicting Marker
Tracking Error. In Proceedings of the 2007 6th IEEE and ACM international
Symposium on Mixed and Augmented Reality (November 13 - 16, 2007). Symposium on
Mixed and Augmented Reality. IEEE Computer Society, Washington, DC, 1-4.
19 - Javanovic, R., & MacKenzie, I. S. 2010. MarkerMouse: Mouse cursor control using
a head-mounted marker, Proceedings of the 12th Conference on Computers Helping
People With Special Needs – ICCHP 2010, pg 49-56, Berlin: Springer.
20 - Apitz, G. and Guimbretière, F. 2004. CrossY: a crossing-based drawing application.

In Proceedings of the 17th Annual ACM Symposium on User interface Software and
Technology (Santa Fe, NM, USA, October 24 - 27, 2004). UIST '04. ACM, New York,
NY, 3-12
21 - http://en.wikipedia.org/wiki/Retroreflector
22- A4Tech http://www.a4tech.com/product.asp?cid=77&scid=160&id=593

88

Report

Uploaded by

Document Information

Original Description:

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Report

Uploaded by

Copyright:

Available Formats

COMSATS Institute of Information Technology, Islamabad

MANIPULATION OF MICROSOFT POWERPOINT ®

Supervisor: Dr. Muhammad Tahir

Bachelor of Computer Science (2006-2011)

Aniqa Hafeez Mohsin Yazdanie

SRS 14 April,2010 First SRS Document 0.1

Detailed design 14 May,2010 Detailed Design Document 0.5

Final draft 20 Dec,2010 Final Documentation 1.0

4.2.8 INTERFACES ........................................................................................................................... 41

9 CONCLUSION AND FUTURE WORK……………………………….……………..……..83

TUI is defined as:-

Development Environment:- Visual Studio 2008

1.2 RELEVANCE TO COURSE MODULE

1.3 PROJECT BACKGROUND

1.4 LITERATURE REVIEW

Literature review helped us in knowing about Augmented Reality, AR markers, ARToolKit,

1.4.1 What is Augmented Reality (AR)?

Figure 1.1 Left: AR application using AR marker and human

1.4.2 WHAT IS TANGIBLE USER INTERFACE (TUI)?

1.4.3 WHAT IS TANGIBLE AUGMENTED REALITY (TAR)?

1.4.4 WHAT IS GESTURE-BASED INTERACTION?

1.4.5 WHAT IS AN AR MARKER?

These are some of the AR markers with different patterns:-

Kanji Sample 1 Sample 2 Hiro

Figure 1.2: ARToolKit Markers

We can make custom markers, keeping in mind following things [12]:

• Patterns should not be very complex.

• Black part of the pattern should be anti reflective material.

1.4.6 WHAT IS ARTOOLKIT?

1.4.6.1 HOW ARTOOLKIT WORKS?

Figure 1.3: ARToolKit working Process stepwise

Figure 1.4 Execution of ARToolKit

The marker’s tracking challenges in ARToolKit include:

• Camera and marker’s orientation

1.4.7 PAPERS REVIEWED

We have reviewed research papers on human computer interaction, gestures, augmented

 Measuring ARToolKit Accuracy in Long Distance Tracking Experiments [14]

 Design of halftone-based AR markers under infrared detection [15]

 Manipulating and annotating slides in a multi-display environment [16]

 Image Manipulation through Gestures [17]

 Method for Predicting Marker Tracking Error [18]

 Mouse cursor control using a head-mounted marker [19]

1.5 METHODOLOGY AND SOFTWARE LIFE CYCLE

2.1 PROBLEM DEFINITION

2.2 DELIVERABLES AND DEVELOPMENT REQUIREMENTS

Following are the deliverables:

2.2.1 DOCUMENTATION DELIVERABLES

2.2.2 APPLICATION REQUIREMENTS

• Windows operating system.

2.2.3 COMMON COMPONENT DEVELOPMENT REQUIREMENTS

• Smart Draw (for UML 2.0).

• Microsoft Office 2007 (for Report writing and presentation).

• Visual Studio 2008

2.2.4 HARDWARE REQUIREMENTS

• Desktop or Laptop System.

 Processor Intel Core2Duo 2.0 GHz

 HDD minimum 40GB.

• Web camera (USB).

3.1 BRIEF OVERVIEW

By using AR Markers, we have performed following operations:

 Go to Specific Slide (Next, Previous)

System : HP Pavilion DV5 Core2 Duo P8400,

Intel Graphics Card.