A Framework for Context-Aware Digital Signage

Ichiro Satoh
National Institute of Informatics 2-1-2 Hitotsubashi, Chiyoda-ku, Tokyo 101-8430, Japan ichiro@nii.ac.jp

Abstract. A framework for building and operating context-aware multimedia content on digital signage in public/private spaces is presented. It enables active and multimedia content to be composed of mobile agents, which are selfcontained programmable entities that can travel from computer to computer and provide multimedia content for advertising or user-assistant services to users by using their own programs. The framework automatically deploys their agents at computers close to their current positions to provide advertising or annotations on objects or users. To demonstrate the utility of the framework, we present a user-assistant that enables shopping with digital signage.

1 Introduction
Active media have been expected to play an important role in advertising, navigation, assistance, or entertainment. For example, modern out-of-home (OOH) advertising has focused on marketing to consumers according to their contexts, e.g., when and where they are on the go in public places, in-transit, or waiting. Ambient services are services that are related to the surrounding physical environment of the user, and can be considered to be a form of location-based services. For example, active media should be selected and adapted to according to the users context. Different kinds of ambient shopping services have been developed to aid users including brokerage services, navigational services, comparison shopping, product searches, location-based auctions, wireless advertising, and even information services where a user can point a device at a particular product or shop to nd out more about it [2,4,5,7,9]. Active media may be useful for advertising products at retailers and public spaces, but they should be used for manufacturing and distributing these products and assisting customers while they are using them and after they have bought them. This paper presents a framework for building and managing active media in largescale ambient computing environments. The framework monitors contextual information in the real world by means of sensing systems and selects and adapts active media according to changes in the real world. It provides massive users with active media at massive ubiquitous and mobile computing devices in their personalized forms according to their contexts. It enables us to easily build and provide a variety of active media, including interactive content. The framework enables creators, who may have no professional knowledge about active media, to compose active media from active and programmable contents. Active media have the ability to monitor and response to user behaviors, because advertising or user-assistant services provided as active media should adapt to the user proles.
N. Zhong et al. (Eds.): AMT 2011, LNCS 6890, pp. 251262, 2011. c Springer-Verlag Berlin Heidelberg 2011

252

I. Satoh

This framework introduces the notion of mobile agents, where mobile agents are autonomous and self-contained programmable entities that can travel from computer to computer under their own control. It assumes that each agent is spatially bound to its target object or space in the sense that it is deployed at a computer close to the current location of the target. Agents are digital representations of physical objects or places and dene and provide active media for advertising or user-assistant active media to users inside them. No physical objects, including legacy appliances, may specify their attributes or interact with one another, because of limited resources. No applications can directly interact with people or physical objects but they can with their agents. Their agents interact with one another, on behalf physical objects. Many researchers have attempted to work on active media, but most of this has been on conceptual or prototype-level steps. There is a serious gap between conceptual or prototype-level attempts and real systems. This paper addresses the implementation and evaluation of an active media system in real spaces with real users for real applications. active media are provided in widespread building and city areas so that they cannot be managed by using traditional approaches, such as those that are centralized and topdown. However, most existing attempts to achieve active media have been implemented in an ad-hoc and centralized manner

2 Basic Approach
This framework enables active media to be implemented as mobile agents, which are active software and be dynamically deployed at computers close to users according to contexts in the real world, e.g., the locations of users and physical objects obtained by using locating systems. It has several advantages in implementing active media. Large-scale context-aware systems need to be managed in a non-centralized manner so that they can be supported. Mobile agents can be managed without any centralized servers. Each mobile agent is a programmable entity with data storage. Therefore, all mobile agent-based active media can dene programs to play their visual/audio content and interact with users. Therefore, the framework itself is independent of application-specic tasks and provides multiple kinds of active media, because such tasks are performed within mobile agents. After a mobile agent arrives at its destination, it can continue work without losing the results of working, e.g., the content of instance variables in the agents program, at the source computers. Therefore, users can continue to watch or listen to active media from computers close to their current positions, even when the users move from location to location. The framework deploys and executes mobile agents bound to physical entities at computers near the position of the user instead of at remote servers. As a result, mobile agent-based active media can directly interact with users, where RPC-based approaches, which other existing approaches have often been based on, must have network latency between computers and remote servers. Mobile and ubiquitous computers often only have limited resources, such as restricted levels of CPU power and amounts of memory. Mobile agents can help to conserve these limited resources, since each agent needs to

A Framework for Context-Aware Digital Signage

253

be present at the computer only while the computer needs the active media provided by that agent. Nevertheless, mobile agent technology itself does not support any active media or context-awareness. The framework offers several functions to enable us to easily build and operate context-awareness active media. It manages location-sensing systems and deploys mobile agents for active media at appropriate computers according to the locations of target users. Furthermore, the activation of active media is classied into three patterns according to the time services are activated. A service is provided in a place, i) before its target is at the location, ii) while its target is at the place, or iii) after its target has left the location.

3 Design and Implementation

The framework consists of two parts: (1) mobile agents and (2) location information servers, called LISs (Fig. 1). The former offers application-specic active media, which are attached to physical entities and places, as collections of mobile agents. The latter provides a layer of indirection between the underlying locating sensing systems and mobile agents. Each LIS manages more than one sensor and provides the agents with up-to-date information on the state of the real world, such as the locations of people, places, and things, and the destinations that the agents should migrate themselves to.

Active media agent

Runtime system

Active media agent

Runtime system Runtime system

Agent Active media agent migration

Runtime system

Agent host Spot 1

Agent host Spot 2

Agent host Spot 3

Agent host Spot 4

Location-sensing system (Proximity)

Location-sensing system (Proximity)

Location-sensing system (Lateration)

Abstraction Filter

Abstraction Filter

communication

Abstraction Filter

Contextual event manager Host Agent Event information information database database dispatcher LIS Peer-to-peer

Contextual event manager Host Agent Event information information database database dispatcher LIS

Fig. 1. Architecture

254

I. Satoh

3.1 Location Information Server Each LIS can run on a stationary or mobile computer. It manages multiple sensors that detect the presence of tags and maintains up-to-date information on the identities of tags that are within the zone of coverage of its sensors. This is achieved by polling sensors or receiving the events issued by the sensors themselves. An LIS does not require any knowledge of other LISs. To conceal the differences between the underlying locating systems, each LIS maps low-level positional information from each of the locating systems into information in a symbolic model of location. An LIS represents an entitys location, called a spot, e.g., space of a few feet, which distinguishes one or more portions of a room or building. The current implementation assumes that RFID tags that periodically broadcast beacons (e.g., every second), including the identiers of the tags, are attached to people and physical entities and each spot has an RFID reader that can receive the identiers of tags within its range of coverage (1-20 meters). Therefore, it can detect the presence (or absence) of a tag attached to a person or physical entity in a spot. When an LIS detects a new tag in a spot, it multicasts a query that contains the identity of the new tag and its own network address to all the agent hosts in its current sub-network. It then waits for reply messages from the agent hosts. Here, there are two possible scenarios: the tag may be attached to an agent host or the tag may be attached to a person, place, or thing other than the agent host. In the rst, the newly arriving agent host will send its network address and device prole to the LIS; the prole describes the capabilities of the agent host, e.g., its input devices and screen size. After the LIS receives the reply, it stores the prole in its database and forwards the prole to all agent hosts within the spot. In the second, agent hosts that have agents tied to the tag will send their network addresses and the requirements of acceptable agents to the LIS; the requirements of each agent specify the capabilities of the agent hosts that the agent can visit and provide its active media at. If the LIS does not have any reply messages from the agent hosts, it can multicasts a query message to other LISs. When there are multiple candidate destinations, each of the agents that is tied to a tag can select one destination on the basis of the proles of the destinations. When the absence of a tag is detected in a spot, each LIS multicasts a message with the identier of the tag and the identier of the spot to all agent hosts in its current sub-network. 3.2 Agent Host Each agent host has two forms of functionality: one is for advertising its capabilities and the other for executing and migrating mobile agents. When a host receives a query message with the identier of a newly arriving tag from an LIS, it replies with one of the following three responses: (i) if the identier in the message is identical to the identier of the tag to which it is attached, it returns prole information on its capabilities to the LIS; (ii) if one of agents running on it is tied to the tag, it returns its network address and the requirements of the agent; and (iii) if neither of the above cases applies, it ignores the message.

A Framework for Context-Aware Digital Signage

255

Each agent host is responsible for executing and migrating agents to other runtime systems running on different computers through a TCP channel using mobile-agent technology (Fig. 3). It is built on the Java virtual machine (Java VM) version 1.5 or later versions, which conceals differences between the platform architectures of the source and destination computers. It governs all the agents inside it and maintains the life-cycle state of each agent. When the life-cycle state of an agent changes, e.g., when it is created, terminates, or migrates to another runtime system, its current runtime system issues specic events to the agent. Figure 2 shows that, when a product is carried by a customer, a mobile agent bound to the product is migrated from computer to computer.
Agent migration Agent migration Agent migration

Product

Production information

Item information

Advertising

How to use

Factory

Wholesaler

Retailer

Endconsumer

Fig. 2. Forwarding agents to digital signage when user moves

When an agent is transferred over the network, not only its code but also its state is transformed into a bitstream by using Javas object serialization package and then the bit stream is transferred to the destination. Since the package does not support the capturing of stack frames of threads, when an agent is deployed at another computer, its runtime system propagates certain events to instruct it to stop its active threads. Arriving agents may explicitly have to acquire various resources, e.g., video and sound, or release previously acquired resources. 3.3 Mobile Agent-Based Active Media Each agent is attached to at most one visitor and maintains his/her preference information and programs that provide annotation and navigation to him/her (Fig. 3). Each agent keeps the identier of the tag attached to its visitor. To support user/locationdependent active media, each agent is dynamically assembled from the content and user-preference parts. Content part: This part is responsible for selecting and playing annotations according to the current spot and route in addition to the information stored in the user-preference part and it plays the content in the personalized form of its user. It is dened as a set of content-selection function and programs to play the selected content. The content part contains a content-selection function and a content player.

256

I. Satoh

Mobile agent-based ambient media

Ambient media content

User-preference part
Knowledge Interests RFID Tag ID Content selection function Agent state manager

Annotation part
Content player program

External ambient media content

Built-in service APIs

Agent lifecycle event dispatcher

Agent execution Agent migration manager manager

Agent runtime system

Java VM / OS / Hardware

Fig. 3. Architecture of runtime system for service-provider agent

Content-selection function: This function maps more than one argument, e.g., the current spot, the users selected route, and the number of times he/she has visited the spot into a URL referring to the annotative content. The content can be stored in the agent, the current runtime system, or external http servers. That is, each agent can carry a set of its content, play the selected content at its destinations, directly play the content stored at its destinations, or download and play the content stored in Web servers on the Internet. The current implementation can divide this part into three sub-parts: opening, annotations, and closing, which are played in turn. Content player: Annotation content varies, e.g., it can be text, image, video, or sound. The annotation part denes programs to play this content. The current implementation supports (rich) text data, html, image data, e.g., JPEG and GIF, video data, e.g., animation GIF and MPEG, and sound data, e.g., WAV and MP3. The format for content is specied in an MIME-based attribute description. Since the annotation part is dened as Java-based general-purpose programs, we can easily dene interactions between visitors and agents. User-preference part. This is responsible for maintaining information about a visitor. In fact, it is almost impossible to accurately infer what a visitor knows or is interested in from data that have been measured by sensing systems. Instead, the current implementation assumes that administrators will explicitly ask visitors about their knowledge and interests and manually input the information into this part. Nevertheless, it is still possible to make an educated guess with some probability as to what a visitor may be interested in, if we know which spots he/she has visited, how many he/she has visited, and how long he/she has visited. Each agent has a mechanism to automatically record the identiers, the number of visits to, and the length of stays at spots by visitors.

A Framework for Context-Aware Digital Signage

257

3.4 Current Status This section describes the current implementation of our system. It was implemented using Suns Java Developer Kit version 1.5 or later versions. Support for location-sensing systems: The current implementation supports two commercial tracking systems. The rst is the Spider active RFID tag system, which is a typical example of proximity-based tracking. It provides active RF-tags to users. Each tag has a unique identier that periodically emits an RF-beacon (every second) that conveys an identier within a range of 1-20 meters. The second system is the Aeroscout positioning system, which consists of four or more readers located in a room. These readers can measure differences in the arrival times of WiFi-based RF-pulses emitted from tags and they estimate the positions of the tags from multiple measurements of the distance between the readers and tags; these measurement units correspond to about two meters. Security and privacy: The framework only maintains per-user prole information within those agents that are bound to the user. It promote the movement of such agents to appropriate hosts near the user in response to the users movements. Thus, the agents do not leak prole information on their users to other parties and they can interact with their mobile users in personalized form that has been adapted to respective, individual users. The runtime system can encrypt agents to be encrypted before migrating them over a network and then decrypt them after they arrive at their destination. Moreover, since each mobile agent is just a programmable entity, it can explicitly encrypt its particular elds and migrate itself with these elds and its own cryptographic procedure. The Java virtual machine can explicitly restrict agents to only access specied resources to protect hosts from malicious agents. Although the current implementation cannot protect agents from malicious hosts, the runtime system supports some authentication mechanisms for agent migration so that each agent host can only send agents to and only receive agents from the trusted hosts. Performance evaluation: Although the current implementation was not built for performance, we measured the cost of migrating a null agent (a 5-KB agent, zip-compressed) and an annotation agent (1.2-MB agent, zip-compressed) from a source host to a destination host that was recommended by the LISs. The latency of discovering and instructing an agent attached to a tag after the CDS had detected the presence of the tag was 420 ms and the respective cost of migrating the null and annotation agent between two hosts over a TCP connection was 38 ms and 480 ms. This evaluation was carried out with three computers (Intel Core 2 Duo 2 GHz with Windows XP Professional and JDK 1.5) connected via a Fast Ethernet. This cost is reasonable for migrating agents between computers that follow visitors moving between exhibits.

4 Applications
We experimented on and evaluated mobile agent-based active media for appliances, e.g., electric lights. These were unique to other existing active media because they did not support advertising of their target appliances but assisted users in controlling and

258

I. Satoh

disposing of them. We attached an RFID tag to an electric light and provided a mobile agent as active media for the light. The active media were attached to their target item and were deployed at computers close to the current position of the item. The current implementation assumed that an agent to manage the active media for their target appliance was created when the appliance was shipped from the factory. 4.1 Location-Aware Active Media Since an agent denes programs to display three kinds of active media content inside it, it selects them according to their spaces. It supports the lifecycles of the items from shipment, showcasing, assembly, use through to disposal. In warehouse: While the light is in a warehouse, its agent is deployed at a computer in the warehouse. It noties a server in the warehouse of its specication, e.g., its product number, serial number, date of manufacture, size, and weight of it. In store: While the light is being showcased in a store, its agent is deployed at a computer close its target object, which displays advertising media to attract purchases by customers who visit the store. Figures 4 a) and b) have two images maintained in the agent that display the price, product number, and manufactures name on the current computer. In house: When a light is bought and transferred to the house of its buyer, its agent migrates to a computer in the house and provides instructions on how it should be assembled. Figure 4 c) has the active media for advice on assembly. The agent also advises how it is to be used as shown in Fig. 4 d). When it is disposed of, the agent presents its active media to give advice on disposal. Figure 4 e) has an image that illustrates how the appliance is to be disposed of. 4.2 Appliance Control through Active Media We can dene agents that control appliances, which may not have any network interfaces. In a previous project [8], we developed a generic server to control power outlets through a commercial protocol called X10. In both the approaches we describe here, the lights are controlled by switching their power sources on or off through the X10 protocol, which involves a user-aware automatic controller and a remote controller. User-aware automatic controller: The rst can autonomously turn room lights on whenever a user with a tagged user is sufciently close to them. The agent attached to the light can also work as our X10-based servers client and runs on a stationary agent host in the room. When a tagged user approaches the light, an LIS in the room detects the presence of his/her tag in the cell that contains the light. The LIS then moves the agent that is bound to his/her tag to the agent host on which the lights agent is running. The users agent then requests the lights agent to turn the light on through inter-agent communication. Remote controller: The second allows us to use a PDA to remotely control nearby lights. Place-bound controller agents, which can communicate with X10-base servers to switch lights on or off, are attached to locations with room lights in this system.

A Framework for Context-Aware Digital Signage

a) In-store ambient media b) In-store ambient media

259

c) In-house ambient media for assembly guide

d) In-house ambient media for using guide

e) In-house ambient media for disposal guide

Fig. 4. Active media for appliance

Each user has a tagged PDA, which supports the agent host with WindowsCE and a wireless LAN interface. When a user with a PDA visits a cell that contains a light, the framework moves a controller agent to the agent host of the visiting PDA. The agent, now running on the PDA, displays a graphical user interface to control the light. When the user leaves that location, the agent automatically closes its user interface and returns to its home host.

5 Related Work
There have been several active media with the aim of enabling users to watch/listen to context-aware information, e.g., annotation about exhibits at the right time and in the right place. Watson et al. [14] have proposed the term u-commerce (or ubiquitous commerce), which is dened as the use of ubiquitous networks to support personalized and uninterrupted communications and transactions between a rm and various stakeholders to provide a level of value over, above and beyond traditional commerce. A number of architectures and prototypes of u-commerce systems have been described in the literature [3]. The Shoppers Eye [2] proposed a location-aware service with wirelessly

260

I. Satoh

Desklamp

PDA (Agent Host) Controller Agent RFID-tag attached to a desklamp

RFID-reader

X10 Appliance Module

Fig. 5. Controlling desk lamp from PDA

enabled portable terminals, e.g., PDAs and smart phones. As a shopper travels about, his or her personal terminal transmits messages, which include information about his or her location, shopping goals, preferences, and related purchase history. When this information is received, stores create a customized offer of goods and services. The Impulse project [13] is a PDA-based system whereby customers may add products to a list, indicating preferences such as warranty terms, merchant reputations, availability, time limits for the purchases and preferred price. When a potential customer enters shopping zones, individual stores, or shopping malls, his/her agent engages nearby merchants in a silent exchange seeking items on the list and opens negotiations on the terms of the sale, alerting the shopper if a deal has been agreed on. It is envisaged that the merchant gains valuable information about customers purchasing behavior during the negotiation process. The MyGROCER project [5] provided smart shopping carts, which could automatically scan supermarket products while simultaneously providing valuable information to the customer through a display device, thus creating a fully interactive shopping trip. The Easishop project [4] has used intelligent agents running on mobile phones to support a proximity-based reverse auction service. The PEACH project [10] has developed and evaluated a visitor-guide system for use in public spaces. The system supported PDAs in addition to ambient displays and estimated the locations of visitors by using infrared light and computer vision. The project proposed a system that enabled agents to migrate between computers [6] by displaying an image of an avatar or character corresponding to the agent on remote computers; however, it could not migrate agents themselves to computers. Like the PEACH project, several existing systems have introduced the notion of agent migration, but they have only supported the images of avatars or codes with specied pieces of information, instead of the agents themselves. Therefore, their services have not been dened within their agents independent of their infrastructures, so that they have not been able to customize multiple services while the infrastructures were running, unlike our system. The Virtual Anatomy Assistant Ritchie [15] was an attempt to seamlessly integrate characters into user actions by using 3D-optical tracking systems but it focused

A Framework for Context-Aware Digital Signage

261

on conversations between individual users and agents and did not support any multiple user settings. We also need to discuss differences between the framework presented in this paper and our previous frameworks. We previously presented an approach for deploying mobile agents that were spatially bound to physical places and objects at computers. The agents moved in the places or were close to the objects [11]. However, the approach was not designed for user-navigation, unlike the framework proposed in this paper. We also constructed a location model for ubiquitous computing environments. It represents spatial relationships between physical entities (and places) as containment relationships between their programmable counterpart objects and deployed counterpart objects at computers according to the positions of their target objects or places [12]. This was a general-purpose location-model for context-aware services, but was not an infrastructure for deploying and operating such services.

6 Conclusion
We designed and implemented a context-aware infrastructure to build and manage mobile agent-based active media. It provides users and physical entities with mobile agent-based active media to support and annotate them. Using location-tracking systems, it can migrate active media to stationary or mobile computers near the locations of users and physical entities to which the agents are attached. In addition, in a noncentralized manner to support large-scale context-aware systems, the system is managed. This framework was originally constructed as a part of our ambient computing project for future shopping. The project involves three research sub-projects in addition to the framework presented in this paper: low-carbon emission logistics, RFID-enabled returnable containers in supply chains, and RFID-enabled carbon credit trading. We plan to integrate the framework with these projects. Acknowledgments. This research was supported in part by a grant from the Promotion program for Reducing global Environmental loaD through ICT innovation (PREDICT) made by the Ministry of Internal Affairs and Communications in Japan.

References
1. Cumby, C., Fano, A., Ghani, R., Krema, M.: Building intelligent shopping assistants using individual consumer models. In: Proceedings of International Conference on Intelligent User Interfaces (IUI 2005), pp. 323325. ACM Press, New York (2005) 2. Fano, A.: Shoppers eye: using location-based ltering for a shopping agent in the physical world. In: Proceedings of International Conference on Autonomous Agents, pp. 416421. ACM Press, New York (1998) 3. Galanxhe-Janaqi, H., Nah, F.F.-H.: U-commerce: emerging trends and research issues. Industrial Management and Data Systems 104(9), 744755 (2004) 4. Keegan, S., OHare, G.M.P., OGrady, M.J.: Easishop: Ambient intelligence assists everyday shopping. Information Sciences 178, 588611 (2008) 5. Kourouthanassis, P., Roussos, G.: Developing Consumer-Friendly Pervasive Retail Systems. IEEE Pervasive Computing 2(2), 3239 (2003)

262

I. Satoh

6. Kruppa, M., Kr uger, A.: Performing Physical Object References with Migrating Virtual Characters. In: Maybury, M., Stock, O., Wahlster, W. (eds.) INTETAIN 2005. LNCS (LNAI), vol. 3814, pp. 6473. Springer, Heidelberg (2005) 7. Meschtscherjakov, A., Reitberger, W., Lankes, M., Tscheligi, M.: Enhanced shopping: a dynamic map in a retail store. In: Proceedings of the 10th International Conference on Ubiquitous Computing (UBICOMP 2008), pp. 336339. ACM Press, New York (2008) 8. Nakajima, T., Satoh, I., Aizu, H.: A Virtual Overlay Network for Integrating Home Appliances. In: Proceedings of International Symposium on Applications and the Internet (SAINT 2002), pp. 246253. IEEE Computer Society, Los Alamitos (2002) 9. Reitberger, W., Obermair, C., Ploderer, B., Meschtscherjakov, A., Tscheligi, M.: Enhancing the Shopping Experience with Ambient Displays: A Field Study in a Retail Store. In: Schiele, B., Dey, A.K., Gellersen, H., de Ruyter, B., Tscheligi, M., Wichert, R., Aarts, E., Buchmann, A. (eds.) AmI 2007. LNCS, vol. 4794, pp. 314331. Springer, Heidelberg (2007) 10. Rocchi, C., Stock, O., Zancanaro, M., Kruppa, M., Kruger, A.: The Museum Visit: Generating Seamless Personalized Presentations on Multiple Devices. In: Proceedings of 9th International Conference on Intelligent User Interface, pp. 316318. ACM Press, New York (2004) 11. Satoh, I.: SpatialAgents: Integrating User Mobility and Program Mobility in Ubiquitous Computing Environments. Wireless Communications and Mobile Computing 3(4), 411423 (2003) 12. Satoh, I.: A Location Model for Smart Environment. Pervasive and Mobile Computing 3(2), 158179 (2007) 13. Tewari, G., Youll, J., Maes, P.: Personalized location-based brokering using an agent-based intermediary architecture. Decision Support Systems 34(2), 127137 (2003) 14. Watson, R.E., Pitt, L.F., Berthon, P., Zinkhan, G.M.: U-commerce: expanding the universe of marketing. Journal of the Academy of Marketing Science 30(4), 333347 (2002) 15. Wiendl, V., Dorfm uller-Ulhaas, K., Schulz, N., Andr e, E.: Integrating a Virtual Agent into the Real World: The Virtual Anatomy Assistant Ritchie. In: Pelachaud, C., Martin, J.-C., Andr e, E., Chollet, G., Karpouzis, K., Pel e, D. (eds.) IVA 2007. LNCS (LNAI), vol. 4722, pp. 211224. Springer, Heidelberg (2007)