HE-Book: A Prototype Haptic Interface for Immersive E-Book Reading Experience

Kazi Masudul Alam, Abu Saleh Md Mahfujur Rahman, Abdulmotaleb El Saddik
Multimedia Communications Research Lab University of Ottawa Ottawa, Canada e-mail:{malam@discover,kafi@mcrlab,abed@mcrlab}.uottawa.ca

A BSTRACT This paper presents an intuitive approach of annotation based haptic interaction with traditional digital reading materials such as eBooks. The research targets to bring a multi-sensory interface consisting of perceptual, cognitive and vibrotactile interactions with the digital reading contents. It leverages a previously developed haptic jacket to receive haptic emotive signals wirelessly in the form of patterned vibrations of the actuators in order to pave ways for intimate reading experience for the readers in various eBook platforms. Index Terms: H.5.1 [Multimedia Information Systems]: Artificial, augmented, and virtual realities—; H.5.2 [User Interfaces]: Interaction styles—; I.3.6 [Methodology and Techniques]: Interaction techniques— 1 I NTRODUCTION Reading is a complex cognitive process, which intends to decode meanings from text by relating words with one existing knowledgebase and understanding. According to Dale [9] more use of modalities ensures effective learning. He described our learning phenomenon numerically 10% of what we read, 20% what we hear, 30% of what we see, 50% of what we hear and see, 70% of what we say and 90% of what we both say and do. Though these numerical labels are not widely accepted but the theory, ”Cone of Learning” is more acknowledged in the research communities relating to analysis of the reading behaviours. Our ways of readings are continuously moulded with the advent of new technological innovations, devices and platforms. For example, eBook is continuously challenging the existence of print book and at times eBooks are replacing the former in schools [17]. In our daily life, we are getting more and more explored to electronic materials than printed books as now-a-days most of the advanced hand held mobile devices provide eBook reading facilities [22]. Haptics is considered to be both as independent and supplementary medium of communication channel [18]. Physical contacts are fundamental needs to mental and psychological development and hence their applications in various cognitive applications have attracted attention of many researchers around the world [12]. Recently handheld mobile devices are widely adopting tactile feedbacks in their touch screen based interfaces [5]. With the growing popularity of the haptic sensory feedbacks in the mobile handheld devices [14], the challenge is now to find new avenues to intuitively incorporating the feature into mobile applications so that interactive storytelling becomes more appealing to the users. According to Mangent, ”Haptic perception is of vital importance to reading, and should be duly acknowledged. The reading process and experience of a digital text are greatly affected by the fact

IEEE World Haptics Conference 2011 21-24 June, Istanbul, Turkey 978-1-4577-0298-3/11/$26.00 ©2011 IEEE

that we click and scroll, in contrast to tactilely richer experience when flipping through the pages of a print book” [17]. As today’s user community is widely accepting haptic interfaces; the effect of tactile feedbacks from the reading materials has started to form a new genre of research interest. Recently, many eBook reader devices have introduced tactile as well as sound feedbacks while flipping a page. Motivated by some of these intuitive interaction styles we present a new approach to evaluate suitability of user’s acceptance of tactile feedbacks from eBook reader contents. This interaction will consider not only page flipping interaction but also different contents of the pages. This is particularly significant as tactile feedback has been present in most of the handheld devices now-a-days [14]. In this regard, we present a preliminary prototype exploring the suitability of haptic feedbacks in eBook reading platforms that generates tactile feedbacks when a user reads an annotated part of the eBook content. The eBook reader application can read annotated digital materials e.g. Portable Document Format (PDF) type of eBooks that listens for touch based interactions from the user. When a user interacts with the annotated parts, the system provides distinct vibrotactile effect through a haptic device e.g. wearable haptic jacket [7]. We incorporated manual haptic annotation of the eBook content [19] with different types of haptic stimulations which also includes three from our previous work [21]. As an example, let us suppose that a user is reading paragraphs in a book that describe a terrible sea storm experience. While reading this part of the book using traditional eBook reader, the user is not provided with any immersive experience. Using augmented reality book [4] user can see this scene using some head mounted display or mobile screen but still do not receive any kind of haptic experience. The difference our system can bring to this user experience is that while reading that very eBook sitting on a haptic sofa, wearing a haptic jacket [7] user will receive various vibrotactile haptic feedbacks during the reading process. Such eBook reader facilitates user with immersive reading experience. And this service will not be limited to eBook reading devices like Kobo, Kindle, Sony etc. but will also include mobile phones or desktop eBook reading platforms. Our contribution in this article is three folds. First of all, we propose the annotation based haptic feedback integration in traditional digital reading contents. The haptic interaction is manually defined by analyzing the scenarios of the eBook content. Secondly, we present a generalized framework for such an HCI application that describes the haptic data annotation scheme and various interaction mechanisms with the annotated content. The annotation method used in the prototype allows the creation of separate annotation files, which can also be stored in remote annotation repository. Third, we present a prototype application that is capable of handling touch based interaction and generates haptic rendering to surrounding haptic devices such as a haptic jacket. Usability test results based on the developed prototype have been prepared to show the suitability of such an approach.


[6] have proposed physical behaviourial simulation of a real book in the presentation of a 3D virtual book. In this paper we have introduced a new type of eBook called Haptic E-Book (HE-Book) that annotates eBook contents with various unique vibrotactile signals and later render the signals while user is reading that part of the book. In Section 2 we provide literature survey of existing or closely related applications. Later this idea has been explored in many areas: gaming. various interaction techniques and technologies. A nice work by Harrison explored the history of book evolution upto 2000.2 we describe the graphical user interface (GUI) interaction method in details. When a person looked at the pages using an augmented reality display he or she could see 3D virtual models out of the pages. From these works we get an understanding of the layout of information. written book contents will be transferred to sound wave and will be played in gramophone and subsequently gramophone will evolve to become pocket-sized player. in Section 3. Finally. chemistry. It also supported multiple user immersion in the same scene.Figure 1: High level architecture of the proposed system The remainder of this paper is organized as the following. we present the offline annotation method and a detailed description of the annotation editor. They have discussed semantics of mixed reality book. in Section 3. introduced a new design of dual display for eBook reader and various supported interactions like folding. which handles eBook annotation and annotation retrieval operations. Gupta et al. Several works on augmented books has explored relationship between real and digital book. We conclude the paper in Section 5 and state some possible future work directions. One of the module is Offline Annotation. They have used camera and projector pair to introduce a tracking framework. we present overall management method of various system modules by the interaction controller module. To the best of our knowledge. data etc. when a user opens an annotated eBook using our HE-Book reader software and touches any annotated part. cultural heritage. haptic annotation to augmented reality books can be an interesting genre to explore. MagicBook [4] project was an early attempt of user transportation between reality and virtuality. Welch et al. 368 . In Section 3. Further in Section 4 we describe the implementation issues of the proposed work and present a usability study of the developed prototype. According to those visionaries.4. 3 H APTIC E BOOK S YSTEM The HE-Book system consists of three separate modules and an interaction controller as depicted in Figure 1. Operations of all the modules are centrally monitored and organized by the Interaction Controller. They extended their prototype [15] to support virtual annotation or bookmarking. Last module is Haptic Interaction that handles content to haptic mapping according to the target output device. Later. In this paper we have presented our introductory prototype of the HE-Book. audio-visual augmentation etc. But according to the best of our knowledge. The main goal of this article is to replicate physical metaphors used in dealings to real book [8]. Grasset et al. ease of navigation and importance of annotation in augmented reality book materials. Later. In a long cherished development of mixed reality book. which monitors 3D position of planar pages when user turns book pages back and forth. french visionaries Robida and Uzanne stated the future of books in 1895. images.3 we have illustrated haptic signal generation scheme. In another research. In their work Chen et al. design space and user experience with such type of interfaces [10]. and fanning in their system. This study mainly focused on reading usability. Further. explored integration of projected imagery with physical book in order to introduce tangible interface to multimedia data. More recently. biology etc. For example. in Section 3. In Section 3. Their book can be loaded with various multimedia contents such as video. possible application areas and further development of interaction techniques. [23] extended the concept further to an immersive virtual reality book aimed at medical training for surgery. first we create and store the haptic annotation file of an eBook. haptic annotation to eBook contents is a new idea where a certain amount of research contribution is possible.1. [11]. Next module is Touch Based GUI that is responsible for user finger touch detection and processing. we illustrate various components of our system that facilitates the annotation of haptic in digital reading materials. Workflow of the system is. Card et al. have focused on design and development of mixed reality applications in broader sense. 2 R ELATED S TUDY According to a research article by Harrison [13]. he/she will receive predefined sets of haptic feedback in his/her surrounding haptic devices. flipping. none of them studied the opportunity of tactile feedback or haptics as an augmentation or annotation to the book content.

Fear etc. Our haptic jacket consists of an array of vibrotactile actuators that are placed in different portions of the jacket and their patterned vibration can stimulate touch in the user’s skin. which is editable. the Interaction Controller takes necessary steps to initiate haptic signal rendering to the haptic jacket system as explained in the later section. Granularity level of our annotation ends at Para i. Afterwards the touched screen coordinate is matched with the eBook page paragraphs to obtain the current paragraph that the user is pointing at or touching. When a user touches an annotated part of the HE-Book the standard touch SDK is used to obtain the screen coordinate of the touched device. haptic enabled mobile phones etc. 3. the user is sent a motor vibration to confirm that a valid haptic annotation for the selected paragraph has been found. In our system architecture we have considered Bluetooth as a method of communication between our system and the corresponding haptic device. vibrotactile feedback will start and end in a paragraph level. and Annotation Retriever. of human body. Haptic Signal Generator plays an important role for device specific haptic signal generation. 3. As soon as the user touches a paragraph of the page that has been previously annotated. Such communication can also be extended to other possible personal network communication methods. Vibrotactile actuators communicate sound waves and create funnelling illusion when it comes into physical contacts with skin. 3) an operator will manually select a paragraph of an eBook i.1. 2) is that a HE-Book user can decide to either use the haptic extension system or avoid it. Figure 3: Annotation editor for the eBook document (i. hence it is important to consider device specific configuration. which we call scene or paragraph based annotation. Later the complete XML file will be stored for further usage by the HE-Book reader system.3 Haptic Interaction Figure 2: XML based annotation file for the eBook document. In one of the prototypes.1 Annotation Editor Using a desktop based software system (Fig. PDF content and tag it using various haptic descriptions. neck. Sadness. Hug. this XML parser will return corresponding Haptic block from the XML file.e. 3. For a request of {Page. Tickle and Kiss [20] in our prototype HE-Book system. The eBook reader software continuously monitors the touched screen coordinate interaction. Anger. the eBook reader is deployed in an emulated touch mobile device. haptic sofa. Influenced by our previous works we manually annotated the eBook content with various haptic feedbacks [21] like Touch. 3. shoulder.e. Annotation File. abdomen.e. In order to produce touch feeling the actuators are activated 369 . In a nutshell this is an XML parser. PDF) page through touch based interaction. Frequency and Delay. The GUI further initiates the haptic feedback retrieval process by sending the page ID and the corresponding paragraph ID to the Interaction Controller module. haptic armband.1. Joy. Every Page element is divided in Para which also has ID that denotes the paragraph no. The Interaction Controller coordinates the retrieval operation and receives the appropriate haptic signal for the selected reading material. Pattern. Further. The user can flip pages and scroll the paragraph of the Based on the page and paragraph information of Touch Based GUI. Para} pair. 1) is responsible of searching the annotation file when it is necessary and retrieve the requested XML block. chest etc. Followings are the detail description of these three submodules.3. Interaction Controller gets haptic description from the Annotation Retriever submodule and generates the described haptic signal for the target haptic device. 3. Location means arm.1 Offline Annotation Offile Annotation module is a combination of three submodules: Annotation Editor. Surprise. Under the Para we have another element Haptic which is described using Type.2 Annotation File Advantage of seperate XML based annotation file (Fig. Fun.1. For each paragraph there will be an XML tagging block. As the target haptic device can be heterogeneous such as haptic jacket. Moreover we also extended haptic annotation to the basic emotion sets Love.2 Touch Based GUI The preliminary haptic eBook system works in a touch based device.3 Annotation Retriever This submodule (Fig. TimeUnit. In our prototype system we have used a Bluetooth enabled haptic jacket which has location based array of vibrotactile actuators. For the annotated paragraph of an eBook we create an XML element Page which has attribute ID that denotes the unique number of the page. A series of small actuator motors are placed in a 2D plane in the jacket in a certain manner.

Even though there is no such direct support in the emulator for Bluetooth we developed a bare-bone reimplementation of the android Bluetooth API where we run a Bluetooth signal processor server (telnet to localhost. which was virtually configured with the COM port so that the Bluetooth device can send signals to the haptic jacket. For each tagging an XML block is created which is edited by the human operator for various haptic attributes.2 we present user study of the implemented prototype system to determine the suitability of the proposed system. creating Bluetooth services and connecting to Bluetooth services. We have used a Dell touch screen monitor for our system and added the touch processing part to the ICEPdf.1 Implementation Details In this section we present the details of the implementation issues of our prototype system. As soon as a touch interaction is performed the Interaction Controller coordinates the screen coordinate based paragraph identification of the eBook document. A Bluetooth device was connected with the PCs USB port. it senses whether the haptic jacket system is still rendering an unfinished haptic signal. Later this XML file is stored to be further used by the ICEPdf viewer while reading annotated eBook (Figure 3). In this case we have considered a text file as our candidate reading material. In general. 4. Now in order to use the simulator we call the BluetoothAdapter. An operator can select a paragraph of a PDF file and tag it using our prototype editor of Fig. We have selected two groups of users who already had experienced 370 . In our mobile based prototype. For serial port based Bluetooth communication we have used GPL based BlueCove [1] library which is very handy for J2SE based Bluetooth communication. The jacket used in our prototype adhering to the aforementioned properties is shown in the Figure 4. 3. We also have added XML retrieving facilities to the ICEPdf for annotation searching. We have added tagging window to the system where we have considered various tagging options. The Interaction Controller polls to acquire user touch inputs after certain interval. we needed to use the classes in the emulated packages. The touch based SDK was readily available to determine the screen coordinate and obtaining paragraph ID of the touched text and this was performed using standard selection operation. The Offline Annotation module takes the page ID and the paragraph ID to retrieve the haptic signal associated with the paragraph and returns the signal data to the Interaction Controller. the Interaction Controller removes the haptic signal from the pool of pending haptic signal queue.bluetooth. Then. Instead of using the classes in the package android.4 Interaction Controller The Interaction Controller plays the central role to organize and synchronize necessary work flow in the haptic eBook reading system. Further the Interaction Controller transmits the obtained haptic signal to haptic devices where the signal is rendered and the user feels the appropriated haptic feedback associated with the learning content of the current book. we have used the Android 2. In order to annotate an eBook we have taken PDF as example and developed a PDF annotator modifying the ICEPdf that we have used as viewer also. In order to develop the eBook reader. Our prototype system communicates with a haptic jacket (Figure 4) when any annotated eBook part is touched by the user. For example. we locally build the ICEPdf [2] open source JAVA PDF viewer. Our desktop based test prototype was adequately responsive in a standard Pentium dual core 32-bit machine with 2 GB systems RAM. We used the Bluetooth Serial Port (SPP) devices with full support for VT-100/ANSI terminal emulation.SetContext(this) at some point in our activity/service. In this manner the Interaction Controller can avoid repeated haptic signal generation for the same paragraphs unless it is otherwise desired by the eBook readers. in a defined manner [3][7]. 4 I MPLEMENTATION AND R ESULTS We present the implementation details of the haptic eBook system in Section 4.5 IDE and the primary language was JAVA. The command format is suitable for sending/receiving Bluetooth signal to the haptic jacket’s Parani ESD200 Bluetooth kit. before the Interaction Controller transmits the haptic signal data to the haptic jacket system.1 based mobile emulator to implement the mobile version of the haptic eBook reader. In addition to coordinating the aforementioned task Interaction Controller ensures that the system modules are not blocking or throttling the operation of the other modules.1. 4. The reimplementation supports easy discovery of the devices. One of our prototypes was developed for desktop eBook readers using Netbeans 6. The usability tests took place at the university laboratory in a controlled environment with twenty volunteers of different age groups and academic backgrounds. Figure 5: Usability study of the HE-Book system. 3. port 8199) on our PC to which the emulators will connect through.2 User Study We conducted usability tests to evaluate the user’s quality of experience with our HE-Book system and to justify the suitability of the proposed approach. the Bluetooth emulator server accepts commands on the form: comID paramname paramvalue · · ·.Figure 4: The Haptic jacket. The Interaction Controller employs carrier sensing algorithm to determine the active/idle states of the other modules. Further in Section 4.

Facilitating the design of vibration for handheld devices. [13] B. pages 177–180. [19] A. In IEEE International Conference on Virtual Environments. Barghout. Guimbretiere. 2008. New York. Lastra. K. Archives of Psychology. A. A. Above study considered only one interaction of a user to the prototype system. M. 2005. [20] A. [18] T. and I. pages 1573–1582. Low. Novel Interaction Methods and Techniques. Lewis. Their activity is monitored throughout the experiment and noted for analysis. Schilit. Tactile feedback for mobile interactions. [14] E. 7-8 2009. Research in Reading. J. Likert. pages 1779–1788. and A. May 2009. Eid. J. J. more than 50% users have shown great interest to HE-Book system. Card. and E. Brewster. State. IEEE Computer Society. 1969. Human-Computer Interfaces and Measurement Systems (VECIMS). New York. IEEE. NY. 12(3):22 –35. 2008. pages 19 –23. 2008. In CHI ’08: Proceeding of the twenty-sixth annual SIGCHI conference on Human factors in computing systems. H. editor. In J. ISMAR ’06. M. and W. volume 0. R. We believe that after some round of tests on the same user set will change their view about the system as many of them were introduced first time with haptic feedbacks. [3] A. [6] S. [11] S. still our user study shows that people have a good tendency of accepting such technology. Hong. K. But. Cha. Rahman. In Proceeding of the twenty-sixth annual SIGCHI conference on Human factors in computing systems. 2009. S. and M. Chi. A. Hoggan. [5] S. and C. Fuchs. Grasset. El Saddik. CHI ’08. Mediated social touch: a review of current research and future directions. H. IJsselsteijn. [23] G. IEEE International Workshop on. 2004. A technique for the measurement of attitudes. A. Park. pages 102–105. NY. HumanComputer Interfaces and Measurement Systems. Dunser. Graph. Washington. pages 370–374. Barghout. Poupyrev. In CHI ’05 extended abstracts on Human factors in computing systems. H. pages 99 –102. [16] R.org. pages 228–235. Hwang. J. 2010. [7] J. Navigation techniques for dual-display e-book readers. Springer Berlin / Heidelberg.-L. We are also working on automatic annotation process of the eBook text from which it would be possible to determine the appropriate haptic signals for the reading materials. Ilie. pages 496–502. H. Spatial resolution of vibrotactile perception on the human forearm when exploiting funneling illusion. The universal media book: tracking and augmenting moving surfaces with projected information. Funaro. CA. In the future. A. In Proceedings of ACM CHI. Towles. 2006. ACM. Although initial user feedback has been positive. 1999. Card. M. E. M. they were asked to rate six assertions in Likert Scale [16]. K. In MM ’09: Proceedings of the seventeen ACM international conference on Multimedia. [17] A.icepdf. The users were requested to use the desktop based prototype and were given certain PDF documents to read which were previously annotated with haptic. [4] M. The magicbook . A. The design of a mixedreality book: Is it still a real book? In Mixed and Augmented Reality. Tanaka. http://www. Kato. volume 5611 of Lecture Notes in Computer Science. based on their interaction experience. E-books and the future of reading. Afterwards. pages 1463–1466. F. nov. Gupta and C. M. Immersive electronic books for surgical training. Billinghurst. Hong Kong. [22] B. Harrison. El Saddik. C. Multimedia. [12] A. Hong. ACM. ISMAR 2008. Audio-Visual Methods in Teaching. M.moving seamlessly between reality and virtuality. NY. A. 3book: A scalable 3d virtual book. M. pages 159–162. 5 C ONCLUSION AND F UTURE W ORKS As computers have become more ubiquitous and invisible. Rahman. and C. Price.haptic and who did not. But. the greater is the satisfaction). 2007. S. we plan on exploring more intuitive ways to present haptic signals from the eBook reading materials rather than touching it. S. Rahman. Chen. S. Bridging the gap between virtual and real world by bringing an interpersonal haptic communication system in second life. and A. Haans and W. Steinbach. J. K. The HEBook is a novel endeavour to attempt such a transitional interface for experiencing various haptic responses from the traditional eBook materials. M. In In Extended abstracts of the 2004 conference on Human factors and computing systems (CHI. [9] E. and S. Dale. [15] L. HAVE 2009. H. we need new interfaces that introduce immersive experience and let users easily move between the physical and digital domains. and A. [21] A. The HE-Book supports manual haptic content annotation schemes. 2001. 2009. Jacko. El Saddik. Mackinlay. USA. J. A. 32(1):65 –73. 7th IEEE/ACM International Symposium on. Agrawala. The ratings of the assertions were in the range of 1-5 (the higher the rating. M. 20:32–39. Hwang. Chohan. The reading appliance revolution.. http://www. Brewster. Yang. June 2010. 2008. pages 1135–1136. Russo. 31(4):404 –419. Investigating the effectiveness of tactile feedback for mobile touchscreens. Cha. Authoring edutainment content through video annotations and 3d model augmentation. we find that there is significant number of users who answered neutral or negative to every assertion. 21(3):6 – 8. Virtual Real. USA. Figure 5 shows the user’s responses for each given assertion. In Proceedings of the 5th IEEE and ACM International Symposium on Mixed and Augmented Reality. M. July 2008. Rahman. E. [8] N. June 2010. USA. Welch. CHI ’05. 9(2):149–159. IEEE Comput. Billinghurst. L. 2009. A. S. Cairns. New York. VECIMS 2008. M. D. Turkey. 2005. Cha. Hypertext fiction reading: haptics and immersion. Mangent. Eid. Marshall. El Saddik. May 2000. [2] Open source java pdf. 371 . In Virtual Environments. Dixon. ACM. and A. China. Jaynes. IEEE Computer Society. Los Alamitos. New York. L. In Haptic Audio visual Environments and Games. Hossain. Istanbul. Computer Graphics and Applications. IEEE. El Saddik. and L. Appl. jan. 140:1–55. we continue to improve the interface. R EFERENCES [1] Bluecove. H. ACM. and A. CA. ACM Press. Johnston. USA. Human-Computer Interaction. DC. S. F. S. USA. and J.bluecove. M. 2006. and M. Golovchinsky. Computer. San Jose. ACM Press/Addison-Wesley Publishing Co. Kammerl. 2008. NY. G. Annotating 3d electronic books. B. Brown. [10] R. Hugme: synchronous haptic teleconferencing. USA. 1932. van Dam. Chi. Becker. Rinehart and Winstor. From the study. pages 1095–1098.. Kissme: Bringing virtual events to the real world. USA.org.

Sign up to vote on this title
UsefulNot useful