D4.3.3 FHW Experimental Results and Recommendations v1.0 PDF

D4.3.
3 Experimental Results and Recommendations

2013-10-29
Efstathia Chatzi (FHW) Christopoulos Dimitrios (FHW)
This deliverable is an final progress report for EXPERIMEDIAs embedded experiment focusing on shared, real-time, immersive and interactive cultural and educational and executed by the Foundation of the Hellenic World at its premises at Hellenic Cosmos in Athens. Starting from the more abstract scenario description provided in the earlier D2.1.2, exploiting the architectural blueprint described in the D2.1.3, taking into consideration the methodological guidelines described in D2.1.1 as well as the ethical oversight principles described in D5.1.1, and of course based on the preliminary work described in D4.3.1 and D4.3.2, the document provides an overview of the work that has been done, the data that has been gathered and the final conclusions regarding to the success of the experiment.
www.experimedia.eu
EXPERIMEDIA
Dissemination level: PU
Project acronym EXPERIMEDIA Full title Experiments in live social and networked media experiences Grant agreement number 287966 Funding scheme Large-scale Integrating Project (IP) Work programme topic Objective ICT-2011.1.6 Future Internet Research and Experimentation (FIRE) Project start date 2011-10-01 Project duration 36 months Activity 4 Experiments Workpackage 4.3 FHW driving experiment Deliverable lead organisation FHW Authors Efstathia Hazi (FHW) Dimitrios Christopoulos (FHW) Reviewers Stephen C. Phillips (ITInnov) Version 1.0 Status Final Dissemination level PU: Public Due date PM21 (2013-06-30) Delivery date 2013-10-29
Copyright FHW and other members of the EXPERIMEDIA consortium 2013
EXPERIMEDIA
Contents
1. 2. 3. Executive Summary............................................................................................................................ 6 Introduction ........................................................................................................................................ 7 Background.......................................................................................................................................... 8 3.1. 3.2. 3.3. 3.4. 4. 4.1. 4.2. 4.3. 4.4. 5. 5.1. 5.2. 5.3. 5.4. 6. 6.1. Goals and success evaluation criteria ..................................................................................... 8 Experiences offered before EXPERIMEDIA ..................................................................... 9 The experience examined in the embedded experiment ................................................... 10 Constraints ............................................................................................................................... 10 Definition of agents ................................................................................................................ 11 Live streaming experiment during the show ....................................................................... 11 Augmented reality experiment before the show................................................................. 13 The monitoring component .................................................................................................. 15 Streaming experimental facility ............................................................................................. 17 Augmented reality experimental facility ............................................................................... 17 Social networks experimental facility ................................................................................... 23 Monitoring experimental facility ........................................................................................... 26 Before the beginning of the experiment .............................................................................. 28
Experimental facility architecture ................................................................................................... 11
Experimental facility implementation............................................................................................ 17
Experiment execution ...................................................................................................................... 28 6.1.1. Deploy the facility ............................................................................................................... 28 6.1.2. Familiarize ourselves........................................................................................................... 29 6.1.3. Inform staff and schedule accordingly ............................................................................ 29 6.1.4. Prepare the informed consent forms ............................................................................... 29 6.1.5. 6.1.5. Prepare the questionnaire. ....................................................................................... 29 6.2. Experiment procedure............................................................................................................ 30 6.2.1. Preparatory phase................................................................................................................ 30 6.2.2. Execution phase .................................................................................................................. 31 6.2.3. Data acquisition phase ....................................................................................................... 31 6.2.4. Questionnaires ..................................................................................................................... 31 6.2.5. Focus groups ....................................................................................................................... 31
7.
Data analysis ...................................................................................................................................... 32 7.1. Demographic first analysis: .................................................................................................... 32

3
EXPERIMEDIA
7.1.1. Statistical views of the data ................................................................................................ 32 8. Ethics, privacy, PIA.......................................................................................................................... 36 8.1. Minimum ethical principles ................................................................................................... 36 8.1.1. Doing good .......................................................................................................................... 36 8.1.2. Doing no harm .................................................................................................................... 37 8.1.3. Risk management ................................................................................................................ 37 8.1.4. Consent................................................................................................................................. 37 8.1.5. Confidentiality ..................................................................................................................... 38 8.1.6. Data protection ................................................................................................................... 38 8.2. Ethical oversight principles ................................................................................................... 38 8.2.1. Informed consent ............................................................................................................... 39 8.2.2. Deception ............................................................................................................................. 39 8.2.3. Data collection..................................................................................................................... 39 8.2.4. Withdrawal from the investigation ................................................................................... 40 8.2.5. Observational research ....................................................................................................... 40 8.2.6. Data protection regulation................................................................................................. 40 8.2.7. Consortium partner responsibility .................................................................................... 40 8.3. 9. 9.1. 9.2. PIA ............................................................................................................................................ 41 Evolution and handling of risks ............................................................................................ 42 Current risk registers............................................................................................................... 42 Risks.................................................................................................................................................... 42
9.2.1. Risks for the participants ................................................................................................... 42 9.2.2. Risks for the experiment .................................................................................................... 43 10. Evaluation .......................................................................................................................................... 45 10.1. 10.2. 10.3. Suitability of FIRE technologies ........................................................................................... 45 Impact of FIRE technologies ................................................................................................ 45 Parameters that affected impact ............................................................................................ 46
10.3.1. Ratio of Devices/visitor .................................................................................................... 46 10.3.2. Quality of Wi-Fi signal ....................................................................................................... 46 10.3.3. AR user interface................................................................................................................. 46 10.3.4. Duration and order of AR event ...................................................................................... 47 10.3.5. Number of visitors.............................................................................................................. 47 10.3.6. Social app and web app UI ................................................................................................ 47 10.3.7. Latency versus Quality ....................................................................................................... 47
Copyright FHW and other members of the EXPERIMEDIA consortium 2013 4
EXPERIMEDIA 10.4.
ECC Metric results.................................................................................................................. 49
EXPERIMEDIA
1. Executive Summary
This document presents the working report of the second phase of the 3rd EXERIMEDIA embedded experiment. The document covers all aspects of the experiment, ranging from the purely technical to the purely theoretical ones. More specifically: Section 2 provides a brief introduction to the document, including a summary of the reasoning that led us to split the experiment in two stages. Section 3 highlights relevant elements of the experiment's background in order to improve the readability and completeness of this document, whilst the reader is advised to refer to previous deliverables of the project for further information. Section 4 makes the architecture descriptions of D4.3.2 more detailed and adopts them to the specifics of the first stage of the experiment. The architecture is presented for the two distinct parts of the experimental facility and the information flow between the different components is also shown. Section 5 continues on the same path and explains how these architectures where actually implemented. Moving on to the actual implementation of the experiment, Section 5 describes how the experimental facility was implemented. Section 6 focuses on the actual execution of the experiment and gathering of the data. Using the implemented experimental infrastructure and following the scenario already outlined in D4.3.1 we explain how we invited a number of participants to experience the EXPERIMEDIA extensions to the venue and provide us with their feedback on them. Section 7 presents the gathered data and based on a brief statistical processing reaches conclusions. Evaluations of our work, both internal and independent, are also presented. Section 8 discusses ethics and privacy. The precautions taken with respect to ethics and privacy were already analysed quite rigorously in D4.3.1. We do revisit the subject here for completeness but the reader should be made aware that much of the text of this section is in fact also found in section 6 of D4.3.1, but here it is adapted to the specifics of the second stage of the experiment, as they finally formed. EXPERIMEDIA Dissemination level: PU Risks have been monitored and handled throughout the design, implementation and execution of the experiment, following the guidelines specified in D1.1.2 and using the registers defined in D4.3.1. A complete review of the evolution of the risk registers is provided in Section 10. Appendices include the form used to get informed consent, the form used to track devices, the questionnaire used during the experiment, and the complete listing of the data gathered. We also include a copy of the external evaluators report.
EXPERIMEDIA
2. Introduction
This deliverable is the final progress report for EXPERIMEDIAs embedded experiment focusing on shared, real-time, immersive and interactive cultural and educational data and executed by the Foundation of the Hellenic World at its premises, Hellenic Cosmos in Athens. Starting from the more abstract scenario description provided in the earlier D2.1.2, exploiting the architectural blueprint described in the D2.1.3, taking into consideration the methodological guidelines described in D2.1.1 as well as the ethical oversight principles described in D5.1.1, and of course based on the preliminary work described in D4.3.1 and D4.3.2, the document provides an overview of the work that has been done to this day, the data that has been gathered and the final conclusions regarding to the success of the experiment. Very early in our work towards the execution of this experiment we came to realize some difficulties with the scheduling of different tasks and events in the timeline of the project that could prove problematic. Most notable among them the fact that the final working versions of the technical partners contributions were not up to date with the rest of the central Experiment Content Component as well as the fact that there would not be a chance for us to have feedback from the project reviewers at a mature stage of the work in order to be sure that we are headed in the right direction. Both constitute very serious dangers for the experiment and therefore appropriate action was required. Therefore, in order to navigate away from the aforementioned dangers we decided to split the experiment in two stages with very distinct character and goals. The first stage is aimed to run quickly, provide an early confirmation of the projects methodology and technical approach, put technical components to a practical test and generate feedback for the technical partners who are working on them, gather know-how that will help the new partners who are just joining the project to seek even higher goals, allow the consortium to have a demonstrable output from early on in the project and give us an opportunity to present our approach to experimenting to the project reviewers during the first year review. The second stage is aimed to build on the experiences and know-how of the first stage in order to perform the complete experimental work that was envisaged for the 3rd embedded experiment in the most suitable manner possible. This document reports on the second stage of the experiment, which has just been completed. The document covers all aspects of the experiment, ranging from the purely technical to the purely theoretical ones, and also hints towards the successful implementation and usage of the FIRE technologies in museum venues.
EXPERIMEDIA
3. Background
The experiment's background has already been discussed in D4.3.2, D4.3.1 as well as in D3.1.1, D3.1.2 and D4.3.3 We briefly review here some main elements of it in order to enhance the readability and completeness of the current text and advise the reader to refer to the abovementioned documents for further details.
3.1.
Goals and success evaluation criteria
The embedded experiment involves the development of an experimental facility and its testing with the participation of real users in real settings. Clearly something will be implemented, some data will be gathered and some analysis will happen. Still, a core question remains unanswered and critically subjective: when has the experimenter done enough? In order to remove the subjectivity and provide a clear measure of success, we identified in D4.3.1 a set of goals for the experiment and defined the corresponding objective criteria. Given the fact that four different levels of success are defined, the objective is to achieve a considerable level of success in the first stage of the experiment reported herein and then build on that in the second and final stage of the experiment in pursuit of even higher success. We review the experiment's goals here briefly, together with their corresponding success criteria, as defined in D4.3.1: Goal 1: Be an EXPERIMEDIA test bed (Success) The experiment can be executed. This entails having implemented the experiment architecture, having made all of the included components operational and having been successful in their integration. Know-how has been gathered. This refers to the gathering of know-how related to the further implementation of the embedded experiment.
Goal 2: Explore suitability of FIRE technologies for the field under examination (Success) Identify differentiation between using and not using the FIRE technologies. In other words, we need to establish that there is a substantial difference for the visitors between the conventional experience currently offered and the one that will be offered in the scope of the experiment. Classify the impact of each component as positive, negative or neutral. This is a more specific version of the previous criterion, as here it is not enough to establish that there is a difference. What is also required is a clear indication regarding whether this difference has an impact that QoE of the visitors and if so whether this impact is positive or negative.
Goal 3: Measure impact of FIRE technologies (Moderate success)
EXPERIMEDIA
Quantify and measure QoE. In other words we need to have designed a measure that quantifies QoE and we also need to have applied this to data gathered from the experiment. Correlate measured QoE to utilized FIRE technologies. Moving a step further, to meet this criterion we should be able to identify the contribution of each component in the QoE, so that strategic decisions can be made regarding the directions that warrant further examination.
Goal 4: Identify parameters that affect impact (Exceptional Success) Measure QoE for different parameters. This criterion is met if data gathered when running different instances of the experiment and different QoE values are computed. The compared instances need to be such that a direct comparison related the differences in QoE to differences in QoS of some kind (e.g. bitrate), differences in the design and execution of experiment (e.g. duration of show), differences in demographics etc. Gathered insight for the design of future experiments. This refers to the gathering of insight related to the implementation of future EXPERIMEDIA installations at Hellenic Cosmos, for example in order to run future experiments.
3.2.
Experiences offered before EXPERIMEDIA
The embedded experiment is built around the VR immersion experience offered by the Tholos. When EXPERIMEDIA extensions are not considered, this is offered mainly as a standalone experience that is not combined with any of the other exhibits or services of Hellenic Cosmos. The typical operation of the Tholos and of the service it offers to its visitors may be graphically modelled as in Figure 1. It is easy to see that this is a mainly one-way communication system, as the museum educator controls the system, thus specifying what the Tholos system will render and project to the visitors, while at the same time commenting on it. As a sole exception to this, visitors are able to participate in electronic polls which determine the path that the Educator will follow altering in this way the flow of the presentation in real time. The main reason for this extremely structured and predefined approach is that the museum educator is working with predefined scenarios, i.e. descriptive texts prepared by the FHW experts. These texts provide information on the 3D worlds in a specific order and therefore the tour in the 3D world has to follow the same order, otherwise the museum educator would be unable to provide synchronized information. Before the start of the show there is a brief pre-show which informs the visitors about the characteristics of the VR immersion system and the Tholos Dome, in reality it is used in order to allow the viewers to get comfortable with the dim lighting needed for the Tholos projection. This contains information such as what they are going to experience and general advices in case of dizziness caused by the immersion, what to do in that case and so on. The pre-show is in most cases not related to the content of the show and does not add to the Quality of Experience that is offered to the visitors.
EXPERIMEDIA
Figure 1. Tholos operation without EXPERIMEDIA extensions.
3.3.
The experience examined in the embedded experiment
The first stage of the experiment done in 2012 was a scaled down version due to review and software availability constraints. In this experiment which is designated as the final stage (second stage) all the available software components were tested. This includes the streaming technology from the THOLOS, the live expert web application, the social Facebook web component, the social Facebook mobile component and the Augmented Reality mobile app. Furthermore all the components provided metrics which were recorded and analyzed using the ECC EXPERIMEDIA monitoring component which was setup on a remote LAN PC.
3.4.
Constraints
The main attendants of FHW shows are children and adolescents. In the EXPERIMEDIA experiments only adults will be considered, which created a question regarding the validity and generality of the results. In order to assure that all the testers would be adults the PR department of the museum invited only participants among the friends and the volunteers of the museum at the execution of the experiment. This meant having to organise the experiment when the usual operation of the Hellenic Cosmos was lower and given that the Hellenic Cosmos is closed in August, that was at the beginning of September. So this framework offered us a very restrained time limit in order to analyse the results.
EXPERIMEDIA
4. Experimental facility architecture

Just as interactive exhibits were introduced in museums to accommodate the presumed educational needs and active desires of young audiences, participatory elements in traditional VR shows may draw in audiences for whom creative activities and social connection are preconditions for cultural engagement. Learning is most powerful when it is immersive, multi-sensory, and interactive. The technologies and experiments that are carried out at the Foundation of the Hellenic World are trying to find new ways to harness the power of the new media for traditional VR shows in Digital Domes. The EXPERIMEDIA mode of operation enhances the Dome visitor experience providing three additional activities. Enable remote experts to participate using realtime video stream technologies. The remote experts get a video-audio feed from what is shown in the Tholos, at the same time a video stream from his pc camera is also send back and the expert is seen and heard inside the tholos by the local participants on the Dome screen and audio subsystem. Participants can text-chat with each other during the presentation and with the expert using a dedicated smartphone app which is based on the Facebook API. Before entering the show local Participants are able to use a dedicated Augmented Reality smartphone app to view, manipulate and deepen the visitors' knowledge on specific artifacts which will be shown in the VR show.
Our experiment consists of two parts. The first part takes place before the show and uses augmented reality technologies to provide information and prepare the audience for the show. The second part takes place during the show, inside the dome installation of the Foundation of the Hellenic World, and aims to improve the overall experience of the audience through their interaction with scientists and experts. Below we provide an architectural overview of each part.
4.1.

Definition of agents
Visitors: The adult audience that participate in the interactive shows of the "Tholos" dome theater. We will refer to them as Visitors. Museum educator or simply "educator" is a person that interacts with the audience and controls the navigation through the virtual 3D environment. Experts: One or more scientists, located remotely, that provide additional commentary on the content that is shown during the show and answer the questions of the visitors. We will refer to these scientists as "Experts".
The agents participating in this experiment are
4.2.
Live streaming experiment during the show
Since the Experts are on a remote location, the main motivation behind this experiment is to allow real-time interaction between the Experts and the Visitors. To this end, the experts and the
EXPERIMEDIA
audience must see and hear the same part of the interactive show what is shown in the dome. Additionally, the experts must be able to hear any questions from the audience, and the audience must hear the experts. Figure 2 shows a high level overview of the locations and the desired flow of information between the agents that participate in this experiment.
Figure 2. High-level overview of the locations and the desired information flow between the agents of the experiment. In this particular example two experts participate in the experiment from remote locations.
The actual connectivity and communication between the components is shown in Figure 3. In this figure we can see that the museum educator holds the navigation control, which specifies the content that should be displayed to the visitors. Based on this input, the cluster in the Tholos dome processes the loaded 3D world in order to render the according location and viewpoint and display it to the visitors in the dome. This is the part that was already supported before EXPERIMEDIA and it is in fact the typical scenario for the utilization of the Tholos. With the EXPERIMEDIA extensions, the Tholos system, in addition to the local projections, also forwards the rendered stream (actually a downsized and 2D version of it) to the video stream server, which in turn makes it available to the experts' application. In this way the experts will be aware of the presented content in real-time. The video stream from Tholos is captured from another PC through a video capture card (AVERMEDIA Game Broadcaster HD). At this PC the video is transcoded along with the audio feed from the educators microphone and are transmitted using Adobes Flash Live Encoding to ATOS Server. The Experts will be using ATOS Flash Player to visualize the video stream. Additionally, the museum educator is able to see video-feeds from the experts, as shown in Figure 11. During the experiments these video feeds were projected right at the beginning of the show in order to introduce the experts and after the show for a live question and answer section. This video and audio feed from the experts is also passing through the ATOS Server, where it is transcoded, and it is made accessible to the educator through a regular website. In order to see this website, the adobe flash player is required, a plugin that is available in most modern
EXPERIMEDIA
operating systems. A minor limitation of the system is that since adobe flash is required, the web-based application cannot run on mobile or embedded devices, which usually lack support for this plugin. Nevertheless, the usage of web-technologies like flash in our system makes the video and audio streams easily accessible from multiple computers.
4.3.
Augmented reality experiment before the show
Before entering the show local Participants are able to use a dedicated Augmented Reality smartphone app to view, manipulate and deepen their knowledge on specific artifacts which will be shown in the VR show. A dedicated space just before entering the show was created using markers on exhibition tables. Using augmented reality techniques, these markers can be recognized by an application that runs on commodity mobile devices. The application can then super-impose virtual objects on top of the real ones, by tracking the position and orientation of the markers, or can simply open a website with additional information, depending on the location of the marker. It is worth noting that this experiment does not require the participation of the experts, the visitors will take the additional information directly from their mobile device. This is an significant advantage, because it minimizes the operating expenses. The Augmented Reality (AR) preshow event besides being very interactive and fun, disseminated serious historical information using innovative technologies. It created a link and high anticipation about the VR show because the artifacts seen in the AR app are actually seen later in the show. Therefore acted as an interest hook, capturing the interest of the participants and preparing them mentally for what is to follow. Most importantly it fostered social activities in groups providing a memorable museum experience. The architecture and information flow of this part of the experiment is shown in Figure 4. With our application, the input video feed from the real world is augmented with virtual 3D objects. Alternatively, when a specific marker is detected in the input video feed, the user is redirected automatically to a website that is related to his particular location. The association of the markers with the virtual objects or the websites is performed using a configuration file.
13
EXPERIMEDIA
Figure 3. Flow of information and component diagram for the experiment.
14
EXPERIMEDIA
Figure 4. Flow of information and component diagram for the experiment. Our application configuration can support up to 50 markers and associate them with different 3D objects or links to websites.
4.4.
The monitoring component
Throughout the show a dedicated server was running which hosted the Experiment Content Component (ECC). The ECC consists of a Apache/Tomcat framework which can be interfaced with various components to take metrics and store them in a PostgreSQL database for analysis of Quality of Service (QoS), Quality of Experience (QoE) and Quality of Content (QoC) experiment data. The various components communicate and send the metrics via the RabbitMQ protocol in real time. For this experiment two components were used to gather data related to the video streams and usage of social media. The Social Content Component (SCC) was instantiated on the dedicated server to collect data about the Facebook usage of the visitors, is reported. The SCC returns information about the amount of users, which picture received the most likes and the top comment for each picture shown in the mobile app. It is worth mentioning that the visitors where not exposed to Facebook directly but used it indirectly through the social app from their mobile phones. Using this social app all comments, likes and answers were delegated to a special event page created for the experiment. This event page was then queried by the SCC to return data.
EXPERIMEDIA
The Audio Visual Content Component was the second component which was used in order to collect metrics for the video streaming. During the show there was always one video stream originating from the Tholos and one or more video streams originating from the Experts video camera. The video stream of the Tholos captured the rendered view of the show and transmitted it live over the internet for remote viewing. This video feed was used by the experts in order to participate in the show. Each expert in turn transmitted through his video camera an image of himself back to the Tholos in order to provide an image to be projected onto the surface of the Dome for the live questions and answers part before and after the show. The metric related to this component was the number of video streams and their bit rate.
16
EXPERIMEDIA
5. Experimental facility implementation

Bellow we outline the implementation efforts that were required in order to support this experiment.
5.1.
Streaming experimental facility
The implementation of the second part of the experiment requires streaming the content shown at the dome to multiple remote locations. The interactive content for the dome is produced by a cluster of PCs, as shown in Figure 2. Each PC renders of this cluster renders only one part of the dome. We have augmented this cluster with one additional PC that renders one extra view of the dome. This required a change in the configuration files of our cluster. The video output of this PC is directed to another PC that encodes the content in MPEG4 H.264 format and streams it to the ATOS central server. The h.264 format was chosen among many others because it is one of the most advanced video codecs and it is known to provide state-of-the-art encoding quality. The quality of the streaming experience is highly sensitive to the available bandwidth. Therefore we have experimented with various encoding settings for the video and audio stream, in order to maximize the image quality and the responsiveness of the streaming content. One technical limitation that was imposed from the ATOS media server was that the stream should be at 60fps. A 30fps stream would be perfectly adequate for our purposes and would require less bandwidth but at the time of this experiment we could not use this option. After some experiments with our line, we have settled on a 1440x1080 video resolution for the stream. This is highly depended to the bandwidth of the internet connection in the FHW facilities. It should be noted that the ATOS media server is located in Holland. Therefore some latency is to be expected in the communication. Aside from the network latency, which is to be expected, some amount of latency was also introduced by the h.264 encoder in our streaming PC. Nevertheless, this latency was only a few seconds, and did not pose a serious problem in our experiment. In the future it might be worth investigating methods to reduce this latency even further.
5.2.
Augmented reality experimental facility
The first part of our experiment, as noted before, involves a mobile application that uses augmented reality technologies. The augmented reality framework that was provided to us for testing was the Metaio Mobile SDK. Using this SDK, we have developed our own mobile application. During the implementation of this application we had to make a several design decisions and tests. Bellow we outline these implementation efforts. First we had to decide which mobile platform to support. The provided SDK supports both the Android OS from Google and the Apple iOS. After consideration, we chose to implement our application in the Android OS, because we have found that the development process is more open. Developing and testing on Apple devices required a registration with Apple, something that could take several weeks for corporate entities and could potentially delay our efforts, thus compromise the project. Additionally, the social networking API that is required for another part of the experiment supports only the Android OS. Therefore the application was made for
EXPERIMEDIA
Android, using the Java language and the Eclipse SDK and the Android Development Toolkit. Figure 5 shows a screenshot of the development environment. The exact versions of the software used are Eclipse SDK 3.7.2, ADK 18.0.0.v201203301601-306762. The target Android OS version was 2.3.7 or higher.
Figure 5. The development environment for the mobile application.
After the decision about the Operating System, we had to choose the particular device that was going to be used in our experiment. We performed our first experiments in a "Sony Ericson Live" mobile phone, shown in Figure 6.
Figure 6. Initial tests were performed in this mobile phone (Sony Ericsson Live). The small screen made it difficult to use our application.
Our application was successfully running in this phone, but we have observed that the screen size was rather small, 3.2inches. The small size of the screen was making it hard for the visitors to see the provided information in the form of 3D Objects or Websites. The details in the 3D
EXPERIMEDIA
objects were not apparent and the websites were difficult to navigate and read. Therefore we have used a device with larger screen, the Sony XPERIA P, shown in Figure 7. This phone has a 4.0Inch screen, which makes using the application more comfortable.
Figure 7. The actual device that was used in the experiment (Sony XPERIA P). The 4Inch screen is a good trade-off between usability and portability.
Ideally, we could also use some Tablet PCs running the Android OS, but the larger size makes these devices less portable. We believe a screen around 4-5 inches provides the right balance between usability and portability.
19
EXPERIMEDIA
Figure 8. Our application running on the Android Emulator on a PC. Although the application can run in the emulated device, the actual experiments and testing cannot be performed in the emulator, because augmented reality applications inherently require the video feed from a mobile device. Thus, all of our testing and development for the augment reality experiment was performed on actual mobile devices.
Next, we had to adapt our virtual reality content to the needs of a mobile device. Our 3D models are mostly designed to be displayed by powerful workstations. These workstations are several times faster than a mobile device, so our content had to be simplifies in order to be usable in a mobile phone. After considerable experimentation, we have found that our particular mobile device can show 3D models with up to 20000 triangles. Another limitation was that the model should have only one texture, instead of multiple layers of textures that we use in our workstations. To overcome this limitation, our artists merged the information from all the layers in a single texture. This texture also has the information about the lighting. This process is called "baking" in technical terms. Therefore, we have concluded that after reducing the polygon count to around 20000 triangles and baking all the information to a single texture, the 3D models were usable in our mobile devices. Next, we have performed various experiments with the type of markers that we were going to use in our experiments. The Metaio SDK supports both "marker-less tracking", where the
EXPERIMEDIA
application can track the location and the orientation of an arbitrary image, and tracking with markers, where the application tracks specific markers, as the ones shown in Figure 9.
Figure 9. Example of markers used in the augmented reality application.
In both cases, the 3D model that is superimposed in the video feed should follow the position and the orientation of the image or the marker respectively. Nevertheless, we have found that the tracking with markers instead of images is more robust. In particular, we have found that when using the markers, the tracking is more stable, meaning that the 3D object is actually closely following (tracks) the marker in the video feed, and does not appear to be moving independently, which was the case when using markerless tracking. In particular, the performance of markerless tracking depends on the contents of the actual image that is used for the tracking, something that we have found unacceptable for our application. Furthermore, tracking with markers was less sensitive to lighting conditions, and was working reliably even on environments with relatively low illumination, thus we have used tracking with markers. The markers are integrated on an information sheets that are placed on various locations at the facilities of Hellenic Cosmos. Figure 10 shows an example of such a sheet, along with the application running on the mobile device.
21
EXPERIMEDIA
Figure 10. Left: The augmented reality information sheet. Right: The visitor places the phone over the sheet and observes the 3D object.
Figure 11. The PC screen projected on the THOLOS of the museum educator, communicating with the remotely located Expert.
22
EXPERIMEDIA
Figure 12. Two museum educators guide the audience through a virtual representation of the ancient City of Miletus inside the Dome Installation of FHW. Low light conditions were required for the projectors. One educator controls the navigation with the joystick, while the other one narrates some historical information. On the right side we can see the PC with the video feed from the experts.
Figure 13. Another view of the educators during the experiment.
5.3.
Social networks experimental facility
In order to leverage the power of social media a Facebook based mobile application has been developed that allows visitors to connect using their personal credentials to a dedicated event page. Using this page the visitors can communicate with each other and with the experts using messages throughout the show in parallel with the VR walkthrough. To this end we have used the Social Networking API (SCC) which is provided by our partners. The app was made for Android using the Java language, the Eclipse SDK and the Android Development Toolkit. The target OS was version 2.3.7 and 4.0. Two versions were created to
EXPERIMEDIA
accommodate the proper install on all Android phones as newer phones were not compatible with older versions. We uploaded two installable applications in the form of .apk files to a local server, which was accessible only to the visitors using Wi-Fi and would let them install the app before entering the Tholos although some additional help is required. Figure 14 shows the various screens of the application.
Figure 14. The visual GUI of the mobile SCC application
The application needed a proper Wi-Fi connection and login to the users Facebook account asking him in accordance to the API for permission to post on his behalf into the dedicated EXPERIMEDIA event page which was created for the event. In essence it was a interface for the respecting event page since all the pictures visible to the app UI were posts into this page and all comments and likes real comments and likes on the page. After login the application featured all the pictures posted beforehand into the page, clicking on these pictures the visitors could either like the photo or comment it. Figure 15 shows some comments and remarks created during our official test run.
Figure 15. The comments sections of the mobile application during the final test run.
24
EXPERIMEDIA
Note that both the language used was Greek since all participant were local citizens. The comments (in blue) were created during the time in the Tholos during the actual walk-through and ranged from questions, remarks to funny quotes and jokes indicating the social style and acceptance of the application. In red you could discern the answers of the experts which were able to see in real time everything that was written by the visitors and comment on it. The experts used a specialized web app GUI for log-in into the Facebook event page and watch the walk through remotely form their office. Figure 16 features the expert web application which was available online for the remote experts.
Figure 16. The remote expert web application
Although we got a lot of info regarding user interface issues later on using the questionnaires and by visual examination of the video recordings the application was reliable and an easy to user interface for posting Facebook related information regarding to the show that they were following live. The following figure (Figure 17) shows the visitors interacting with the app during the show at the Tholos.
25
EXPERIMEDIA
Figure 17. Visitors interact with the mobile application during the Tholos show.
5.4.
Monitoring experimental facility
To capture metric data for the dedicated EXPERIMEDIA experiment content component (ECC) was deployed locally on a Linux server which had a static IP internet access in order to be visible by the video streaming server in Holland and the Facebook API network calls from the web app and the mobile app. The ECC framework is general and allows to load, monitor and store the metrics for any EXPERIMEDIA component through separately running client monitors. The client monitors retrieve the data from a device and send it using network protocols to the ECC. Although these monitors could run remotely we choose to run them on the same machine but communicate to the ECC using the static external IP address. Figure 18 shows the ECC service running and detecting both the AVCC monitor client and the SCC monitor client.
Figure 18. The ECC component running on our local server. Copyright FHW and other members of the EXPERIMEDIA consortium 2013 26
EXPERIMEDIA
The server OS was Debian Squeeze and the ECC and monitor clients needed the following software services. RabbitMQ 3.0, PostgrSql 9.2.2, JDK 1.7, Tomcat 7.x, Maven 3.04. Besides the ECC the server also hosted the web page and java code for the expert web GUI and Facebook web app. In order to avoid clashes and port collisions the Facebook web app which needed a older tomcat version 6.x running, a spring tool suite emulator was used which run the whole application along with the corresponding tomcat service in a virtual machine.
27
EXPERIMEDIA
6. Experiment execution
In this section we present the actions taken and procedures followed in order to conduct the experiment.
6.1.
Before the beginning of the experiment
A great deal of effort was required before even approaching the first candidate participant. Specifically, we needed to: deploy the experimental facility, familiarize ourselves with the experimental facility, so that we could provide assistance as required, invite through social media and mouth-to-mouth the persons to participate inform staff about the upcoming experiment and schedule accordingly, prepare the informed consent forms and, prepare the questionnaire.
6.1.1. Deploy the facility

This refers to the actual selection and development of the interest points for which AR services will be offered in a pre-show space (Figure 19) just before entering the Dome presentation. With the help of FHW historians and archaeologists in this phase 4 points were identified that are related to Miletus (which is the topic of the Tholos projection that will be used in the context of EXPERIMEDIA) and for which additional content is available. For 4 points a pure AR service is provided as a 3D object will be superimposed on their camera input. Specifically: Point 1 is a 3D reconstruction of a bed that could be found in the city. Point 2 is a 3D reconstruction of a building that could be found in the city. Point 3 is a physical reconstruction of an ancient ship. Point 4 is coupled with the reconstruction of an amphora.
Figure 19. The pre-show AR exhibition space and visitors interacting with the exhibits.
In the initial test run in 2012 the interest points were scattered around the whole Hellenic Cosmos Museum campus in relevant places. Museum shop, atrium, mathematics exhibition in order to provide a seamless integration with the building in a seamless manner and provide a
EXPERIMEDIA
memorable experience. Our first test resulted that it was very difficult and time consuming for visitors to find them while scattered around, and they did not fulfil their intended use of providing an easy to digest and natural experience with social interaction. This is why all points were gathered in a pre-show room to be experienced right before the show in an easily accessible manner providing also a place for social interaction.
6.1.2. Familiarize ourselves

Regarding the augmented reality component that was the first stage of the experiment familiarization with the software was relatively easy again as in 2012. The related application is quite easy to use (once it is turned on all a user has to do is point to the card) so not much training was needed. Though for both the second stage of the experiment and the Augmentned Reality Component we had to provide both the museum educators and the experts with effective operational guidelines both on the operation of the devices as well as set the devices up for those cases that brought their own devices. We needed at least an hour to prepare 8 devices: install the components, connect to the Wi-Fi of the venue and test the device. The testing group was using 4 devices (already set up) provided by us and 14 devices of their own so at least 2 hours were needed simply as a technical preparation for the devices. For those reasons we decided to have a meeting the day before with the persons participating in order to set up their personal devices that they would be used in the experiment also.
6.1.3. Inform staff and schedule accordingly

When it comes to the augmented reality component no scheduling was necessary. For the Tholos, on the other hand, we needed to find free time in the Tholos schedule, book it for EXPERIMEDIA, arrange with ATOS so that streaming would be available, arrange with the museum educators and make sure the experts were also available a the same time. We have come to realize that the expert has to be a person that knows the content much better than the museum educators (who are also historians and archaeologists) for this to make sense already from stage one of the experiment back in 2012 (D4.3.2 par. 6.1.3) We decided for this second stage to both alter the scenario and book ahead. Given the fact that the Tholos projection was about the Ancient Miletus we decided to use as the expert, the lead architect of the 3D reconstruction group that had designed the virtual city of Miletus and not only book him ahead but also ask him to acquire the latest information from the architects and archaeologists of the physical excavation of the Ancient Miletus.
6.1.4. Prepare the informed consent forms

The informed consent form was prepared based on a sample that was provided by the EAB. It can be found (in Greek) in Appendix A.
6.1.5. 6.1.5. Prepare the questionnaire.

The questionnaire was prepared based on the ethical and privacy guidelines that were agreed upon. Therefore only multiple choice questions are included. The questions examine the participants views regarding the clarity of the image, the relevance of the content to the point of
EXPERIMEDIA
activation, the participants interest in the presented content, the added educational value and the fun they had using the application. It also asks whether the participants would accept to pay a fee of 1 euro to have access to this application, or to similar application with more points of interest. All the above are asked both generally and also separately for each point of interest. The questionnaire can be found (in Greek) in Appendix C
6.2.
Experiment procedure
6.2.1. Preparatory phase

In general it took the technical and programming team in direct communication with the partners developing the actual components about 2 man-months (June-September) of installing the components, making sure of using the correct versions, testing them etc. One consideration was the selection of participants. This was generally done by our Public Relations department one week before the experiment started, so that we would have enough time to find an adequate number of participants. In fact candidate participants were asked beforehand whether they would be willing to participate at the experiment, complete the questionnaire and some of them participate to the focus group. As agreed, no minors were approached. Another consideration was how to properly use the devices inside the Tholos in a structured way that would not harm or annoy the rest of the immersion experience. Having discussed this with our department of Historical Research as well as the museum educators we had also to come up with a scenario on when the expert should appear (before a trigger point e.g. the theatre) and how to include this in the normal educational scenario. This way the participants would use the device after having been asked by the museum educator, send the questions in different point of the guide, then receive the answers at certain times all together (so no annoying message receptions that could distract the participants at different times during the guide), as well as a nicely timed appearance of the expert at the given trigger point, that would provide more information, listen to the questions of the participants in real - time and reply to them as well in real-time all in a given structured scenario that the museum guide could follow. This procedure of scenario development also happened before the experiment and took about a week for everything to prepare. The day of the experiment, the experiments goals, the overall procedure, their role in the experiment, the nature of the gathered data, the handling of the data etc were explained in detail to every candidate orally, and then the written consent form was presented. We were available to answer any additional questions posed, but there were no such incidents. The actual experiment run over 2 days (including the setting up of the personal devices of the participants in day 1) using 4 devices belonging to FHW and 14 devices belonging to the participants so we worked with a total of 18 participants. It took approximately 2 hours for the overall experiment. The filling of the questionnaires took about 30m and the focus group with 6 participants took another 30-40 minutes.
EXPERIMEDIA
We also found out that we had to set up an experiment team consisting of different disciplines in order to implement the experiment. The team consisted of one experiment manager coordinating all the involved departments in the FHW, the technical and programming team, the Public Relations team, the maintenance and IT team, the Museum Educators Team and the Historical Documentation Team.
6.2.2. Execution phase

This is the phase in which the participating visitors interact with the Tholos as well as the physical exhibitions of the Hellenic Cosmos and use the different components. The participants without their own devices were required to sign for the 4 smart devices we provided them with. This documentation was not combined with any of the experiments data and was only maintained until the end of the tour, when the device was returned. The form signed for the temporary provision of the smart device is presented in Appendix B. The brochure described in D4.3.1 was not created, as we chose to physically place the markers at the entrance of the Tholos.
6.2.3. Data acquisition phase

All the components of the experiment were executed, Visitors were the only participants. We used questionnaires and focus groups to poll them for information.
6.2.4. Questionnaires
The questionnaire used has already been presented.
6.2.5. Focus groups

Focus groups differ from conventional survey methods such as questionnaires because they use insight instead of rules, and have a social and not individual orientation. They also have the benefit of translating the whole experience to words and feelings instead of numbers, which allows the participating users to be much more expressive in their feedback. On the low side, the unstructured nature makes it is considerably harder to process and analyze information acquired in this way for these reason we tried to keep the questions lines related to the questionnaire questions and try to interrelate the results in order to keep a structure to the data analysis. In the first stage of the experiment it proved impossible to hold focus groups. The reason is purely technical: focus groups work well when a large number of participants has first been polled through structured questionnaires and then based on the analysis of these questionnaires the researchers determine some points on which they would like to have additional information and organize focus groups. The small sample of the first stage made it impossible to proceed with the focus group. In the second stage we had a much bigger sample and we were able to organise a focus group at the same day of the experiment. The focus group consisted by 4 people that were of different disciplines relevant to the exhibitions and productions of FHW. So the focus group was consisting of an archaelogist/museologist, a curator, a 3D expert and a Museum pedagogy expert that have participated to the experiment.
EXPERIMEDIA
7. Data analysis
Given the limited time to analyse the data that has been gathered, data analysis is much less rigorous that one might expect from such a complex and ambitious experiment. We do envisage more powerful data analysis to be done at a later version of the deliverable as it is also needed by the venue. The small size of the gathered data allows us to include the complete list in this document. The list can be found in Error! Reference source not found.. Despite the medium number of samples, the number of questions allows a wide range of statistical values, correlations and graphical charts to be produced. We present below some statistics that we find most informative and interesting. Questions are translated roughly and abbreviated. A standard 1-5 Likert scale is used for most questions. The exception is Yes/No questions which are treated as 1/0 values.
7.1.
Demographic first analysis:
The friends and volunteers of the museum are mainly either young professionals or students studying on relevant to the museum activities disciplines (museology, digital archaeology, education, 3D programming and design etc), or professionals of the same disciplines. This of course meant that the testers were a group specifically invited to participate and not just random visitors of the venue, and also that the main demographics of the tester group were: mainly women (78%); high academic background; between 26-40 years of age.
7.1.1. Statistical views of the data

In the tables below we are including the data of the questionnaires interrelated with the comments and data that came out during the focus group for the relevant group of questions Augmented Reality
Question
Q1: Was the device difficult to have on you? Q2: General clarity of images in the applications Q3: Clarity of the bed image Q4: Clarity of the building image Q5: Clarity of the ship image Q6: Clarity of the amphorae image Copyright FHW and other members of the EXPERIMEDIA consortium 2013
Value
1,22 4,22 4,39 4,83 4,94 4,78 32
EXPERIMEDIA
Q7: General interest factor of the content Q8: interest factor of the bed Q9: interest factor of the building Q10: interest factor of the ship Q11: interest factor of the amphorae Q12 : General Educational added value of the application Q13 : Educational added value connected to the bed Q14 : Educational added value connected to the building Q15 : Educational added value connected to the ship Q16 : Educational added value connected to the amphorae Q17 : Was it fun to use the application? Q18: Was it fun to watch the bed? Q19: Was it fun to watch the building? Q20: Was it fun to watch the ship? Q21: Was it fun to watch the amphorae? Q22: Would you pay 1 euro for this service? (1=no, 2=yes)
4,61 4,00 4,50 4,94 4,72 5,00 4,50 4,72 4,78 4,74 4,83 3,22 4,80 4,89 4,85 1,83 1,89 1,89
Q23: Would you pay 1 euro for this service if there were more interest points included? Q24: Would you pay 1 euro for this service if there were more interest points included like the bed?
The focus group revealed that the Augmented Reality was very well received and all agreed that it is a good way to introduce added information and add to the experience. In a different level most participants added that it is an application easily implemented in and before or after a new show or exhibit. Resources-wise it does not require high maintenance, it requires a low percentage of personnel involvement and can easily be recycled for different relevant exhibits. It also does not require a lot of space and can easily fit into (e.g. the frames of exhibit) without changing any previous exhibit design. Also during the first stage (2102) we noticed in the questionnaire data that quite interestingly there are huge differences in the average values for the different points. It is clear that points such as the ship and the amphorae have made a bigger impression than for example the bed and we discussed it in this focus group. The participants indicated two different patterns: The images of the bed and less of the building were considered less detailed than those of the ship and the amphorae (probably because of the rendering angle) The bed was considered to be quite out of context presented on its own without other objects of daily use of the same category.
33
EXPERIMEDIA
Also people would be willing to pay money for such an application. This is probably the safest way to conclude that the augmented reality component did enhance their experience considerably. Although we tried to correlate the quality of the images to the quality of the experience (e.g. in the case of the bed) it looks like the overall experience is not dampened by a not so top quality image. The fun factor is bigger and slight software issues are not diminishing the overall quality of the experience, instead it makes it more fun. Other components
Question Q25: was it clear why the speciifc images were used? (Tholos application) Q26: was there added educational value to the replies of the expert? Q27: Was it fun to use the application inside the Tholos? Q28: Was the quality of the image and of the sound acceptable during the interaction with the expert? Q29: Was it easy to use the Q&A application and its software? Q30: The interaction with the remote expert added value to the experience? Q31: How much distracting was the use of the smart phone during the show? Q32 : Would you like to have permanently in each show a remote expert appearing except the museum guide? Q33: How much would you like to have a permanent service with smart phones in the Tholos? Q34: The quality of the image of the expert was good (yes/no) Q35: The quality of the sound of the expert was good (yes/no) Q36: Was there delay in the reception of the expert's voice. Were you annoyed by the time lapse between the question of the audience and the reply of the expert? Q37 : Which service you liked more in the Tholos (1=Q&A, 2=Expert discussion) Q38 : Would you play 1 euro for the Tholos service? Q39 : Would you pay 1 euro for this service if there have been more interactive services? Value 1,78 1,89 4,61 4,67 4,44 4,67 3,33 3,44 3,89
4,11 4,11 1,56
1,44 1,69 1,72
Focus group discussion confirmed the questionnaire findings that it was quite distracting trying to use the smart device during the Tholos show. Although as mentioned before we tried to
EXPERIMEDIA
introduce a specific scenario in order to structure in a way the times the audience's attention would be distracted and that the questions would be addressed to the expert. The remote expert added value to the information that was given by the local guide and this is an experience that can be repeated but not on a permanent basis (e.g. opening of an exhibit to an experts audience in order to evaluate the quality etc) and not for all audiences. This service according to both the questionnaires and the focus group is something that people would pay for but not so willingly as for the Augmented Reality service. The portability of the device was not an issue everybody considered it easy to move around as was the software once it was installed.
35
EXPERIMEDIA
8. Ethics, privacy, PIA

EXPERIMEDIA conducts research with human participants and is, in particular, interested in human behaviour and experience with Future Internet technologies to understand how to provide meaningful collective experiences to individuals and society. Given that participants in social and networked media research should have confidence in the experimenters, good research will only be possible if there is mutual respect and confidence between experimenters and participants. As some areas of human experience and behaviour may be beyond the reach of experiments, observation or other form of investigation and may raise ethical considerations, EXPERIMEDIA will provide an ethics management process that incorporates ethical and data protection review of experiments. Appropriate management of ethical issues was guaranteed by the project management through a mixture of measures to ensure the right technical, physical and administrative environment. The project identified an ethical issues coordinator, as well as a data protection coordinator, which will be incorporated within the overall project management structure. Additionally, an Ethics Advisory Board (EAB) and Data Protection Board (DPB) have been created. The EAB will advise the EXPERIMEDIA consortium on ethical, privacy and data protection issues. The DPB is responsible for ensuring that EXPERIMEDIA is compliant with data protection requirements and that the technical partners develop a system that considers privacy. These are described in more detail in D5.1.1, which also includes the results of extensive work towards the establishment of guidelines that should be followed in the experiments, in order to ensure compliance with ethical requirements and respect for the privacy of those involved in the experiments. In addition to that, D5.1.2 identifies points that specifically the FHW embedded experiment should consider and D2.1.1 details the Privacy Impact Assessment (PIA) methodology that should be followed by all EXPERIMEDIA experiments. In the following we elaborate on how these have been considered and incorporated specifically in the design of this embedded experiment. Given the overlapping nature of the topics and the documents, there may be some repetition of concepts. We keep it in the text by choice, in order to also maintain the point by point reference to the above documents.
8.1.
Minimum ethical principles
In D5.1.1 a set of ethical principles has been identified for the embedded experiments. They have all been considered in the design of this embedded experiment, as explained in the following.
8.1.1. Doing good

The experiment assesses the added value provided to end users by the extension of the Tholos infrastructure and of the Hellenic Cosmos venue in general via exploitation of the EXPERIMEDIA technology. Therefore it does good both for those participating in the experiment as well as in general.
EXPERIMEDIA
As far as the participants of the experiment are concerned, they had the opportunity to benefit from enhanced services that were previously unavailable. In the more general sense, this experiment will be a first step towards making these technologies a part of the normal operation of the Tholos (remote experts participating) and of the venue (augmented reality points scattered in the venue), so that more people can benefit from them in the future.
8.1.2. Doing no harm

Although people working in EXPERIMEDIA have been monitoring closely the execution of the experiment in all stages in order to analyse every relevant piece of information that becomes available, the actual navigation in the virtual worlds is still performed by the properly trained personnel who do that task in the conventional Tholos shows. Their training and expertise guarantees the quality of the experience that will be provided to all participating user groups. For general public the Tholos is meant to provide a feeling of what it was like to live in another era, which will be achieved for the groups of the experiment as well. For a special case of visitors, though, there is a different goal: the students and pupils that visit the Tholos are mainly meant to receive assistance in their history courses. No harm will be done in this direction either, as these user groups were not considered as eligible participants for the experiment and did not participate. When it comes to the augmented reality component, this merely adds a paper sign to the venue. It makes no difference to those not carrying a specialized device and only offers additional and very relevant information to those who do. Moreover the devices that were not owned by FHW were set up with the consent of the owner and FHW made sure that there was no malicious software installed to the devices to the best of our knowledge.
8.1.3. Risk management

As the experiment design was formulated, and now as the two stages of the experiment are executed, risks are constantly analysed, evaluated and treated, in the same sense as in D1.1.2. In the experiments risk register, two types of risks are identified: risks for the participants and risks for the experiment itself. In the context of ethical oversight of the experiment of course, it is the former that is of core interest. The current instance of the risk register for the participants is displayed in Table 1. As can be seen all identified risks have been treated with the AVOID option. In fact this is a strategic choice for the experiment: risks for the participants will be avoided, even if that moves the risk to the experiment itself. In this manner we can be assured that the participants of the experimenters will not be facing any risks.
8.1.4. Consent
The preparatory phase of the experiment involves the explicit communication of any relevant information to the eligible participants (i.e. what the experiment is about, what it entails, which is their role, the use of their own devices etc). Only those eligible participants that have agreed and have signed a note of informed consent are considered in the experiment. This consent has a
EXPERIMEDIA
predetermined duration of two months. Differently to what was stated in D43.1, this consent is not revocable after the submission of the information. The reason is that all information is fully anonymised from its very creation and therefore it is technically impossible to locate and remove the information provided by a specific person.
8.1.5. Confidentiality
During the experiment only the required data is gathered, this data will only be made available to the individuals that are needed to process that data and no part of this data will be disclosed to any third parties. Gathered data is fully anonymised. All data will be purged after the analysis has been completed and at the latest two months after its gathering.
Figure 1. One of the completed questionnaires. No personal data is listed and all options are multiple choice so that no handwriting is required either
8.1.6. Data protection

Following the review of the experiments plan by the EAB we have reached the concl usion that we do not need to follow a rigorous data protection plan.
8.2.
Ethical oversight principles
D5.1.1 has also produced a more detailed set of ethical principles, more customized to the specifics of EXPERIMEDIA and the embedded experiments. These have also been considered and adopted in the design of the experiment, as seen in the following.
EXPERIMEDIA
8.2.1. Informed consent

All participants are informed of the research objectives and all aspects of the research that might reasonably be expected to influence willingness to participate. We explain all other aspects of the research about which the participants may enquire. This is done in the preparatory phase of the experiment, before participants are asked to join the experiment, so their decision constitutes informed consent. We do make sure that participants do not feel pressured to take part in the experiment. In fact, in order to make sure that we do not put pressure on people to take part in the experiment, we have decided not to set daily goals for the number of participants, so that the people in charge of recruiting do not become extra pushy when the numbers do not add up. There is not and will not be any payments to participants. We have stated in D43.1 that incentive mechanisms used may include the offer of free passes for Hellenic Cosmos exhibitions, which we did implement in the second stage of the experiment. Even if applied, this will not put visitors that accept to participate in any higher risk than that of visitors who do not participate and pay for their passes for the exhibitions. The software was tested on all the devices owned by the FHW and attention was given on the way that it was installed on the personal devices of the participants through a specially appointed server that was running an antivirus protection.
8.2.2. Deception
We never intentionally deceived, misled or withhold information from participants over the purpose and general nature of the investigation.
8.2.3. Data collection

For the second stage of the experiment we did not collect any personal data about the participants. We mentioned in D43.1 that we would collect some personal data about participants during the second stage of the experiment (for example demographics). In which case, we provided participants with any information to complete their understanding of the nature of the research. We discussed with the participants their experience in order to monitor any unforeseen negative effects or misconceptions, in the scope of the focus groups. We since also the first stage adopted a principle of data minimisation: only the necessary information will be collected and processed and the information will be stored only for as long as is necessary. Although it was not possible not to record specific individuals during the second stage experiment, as we needed to keep track of the individuals that have the EXPERIMEDIA devices in their possession, but we kept anonymous the data when the devices were returned at the end of each session of the experiment and did the same for the private devices. User profiles were only stored when the consent of the users was acquired and only for the purpose and lifetime of corresponding experiments session. There will be no commercial exploitation of user profiles.
39
EXPERIMEDIA
8.2.4. Withdrawal from the investigation

We made it plain to participants that they have the right to withdraw from the research at any time, irrespective of whether some incentive has been offered and accepted. The participants will NOT have the right to withdraw retrospectively their consent given and to require that their own data be destroyed, simply because with the fully anonymized data it is not possible to do that. Our EAB assures us that this is Ok.
8.2.5. Observational research

The experiment run in Tholos, which is used by user groups. In order to avoid observing individuals that have not given their consent for this, we run the experiments only with pure groups, i.e. with groups that contain only people that have agreed to participate in the project. In any case observing in the context of this experiment does not include any kind of recording and is limited to observing group dynamics during the show. This is something that is done in the Tholos irrespective of the experiment, as the museum educator that is coordinating the show is always observing the groups response and adapts the navigation accordingly. If the technological choice is made to use audio in order to interact with the experts (the current plan is to post to social networks instead), then only the rooms general sounds will be recorded and not each individual separately. In any case, this will only be allowed for the pure EXPERIMEDIA groups.
8.2.6. Data protection regulation

As expected, we do state that all personal data will be captured and processed according to the applicable data protection provisions, such as Directive 95/46/EC on the protection of individuals with regard to the processing of personal data and on the free movement of such data, including Article 29 Working Party 8/2010 opinion, and Directive 2002/58 on Privacy and Electronic Communications) and the Greek data protection legislation that may be applicable. Having said that, we clarify that this is in fact a void statement for the first stage of the experiment due to the fact that no personal data whatsoever was captured. It remains to be determined whether any personal data will be considered for the second stage of the experiment, as the related questionnaires have not yet been prepared.
8.2.7. Consortium partner responsibility

Be sharing the early version of the experiment description with the consortium we invited our partners to participate in the ethical review of our plans. As the experiment design progressed we updated our partners about our plans. Our partners are of course also invited to monitor the experiment either for the sake of acquiring a better of the operation of the technical components they have provided or simply for their information. Any concern that they may have at that time will be considered and treated accordingly.
40
EXPERIMEDIA
8.3.
PIA
As was shown in D43.1, no further PIA is required. Nevertheless, written consent of a two duration of two months is acquired. Efstathia Chatzi and Dimitris Christopoulos will act as the data controller for the experiment. Since no other personal data is recorded, the data controllers duty is limited to keeping safe the forms of informed consent (which list the participants names) and deleting them at the end of the predefined period.
41
EXPERIMEDIA
9. Risks
At the beginning of the work on the experiment, two risk registers were formulated: risks for the participants and risks for the experiment itself. These were initially reported in D43.1, but were also constantly monitored as a live document, and updated as the project and its environment evolved. In the following we start by presenting how the evolution of the risk registers has affected the work on the experiment to this day, as well as the current instance of the registers with respect the upcoming work.
9.1.
Evolution and handling of risks
In the time that has elapsed since the preparation and finalization of D43.1, and as work on the experiment progressed, the risks associated with it also evolved accordingly. As was clearly stated in D43.2, a firm decision has been made to avoid all risks that concern the experiment participants. This is the main reason that there have not been any changes to that risk register. The experiment's risk register on the other hand has gone through various phases. The most important cause for the evolution of the experiment's risk register was the implementation itself. For example, the D43.1 experiment's register contains risk E2, which refers to the risk of required EXPERIMEDIA components' versions not available on time. As the implementation of the experimental facility progressed it became clear that E2 needed to be partitioned into various distinct items, each one referring to different EXPERIMEDIA components and monitored independently. Thus E2 was reformulated as E5, E6, E7 and E8, each one with low probability and high impact.
9.2.
Current risk registers
9.2.1. Risks for the participants

For the reasons listed above, the current instance of the risk register for the participants is as follows:
Table 1. Risk register for the participants.
ID P1
Description Mobile device damaged during the installation of the mobile application
Probability medium
Impact Proximity high Execution phase
Response avoid
Comment Although consent was given for the use of the personal devices we tried to minimize the risk by installing the software the previous day and test it See point P1
P2
Mobile device not compatible with the EXPERIMEDIA software
high
high
Execution phase
avoid
42
EXPERIMEDIA
P3 Malicious software installed on the mobile device high high Execution phase avoid
After stabilizing the version to be installed we uploaded to a dedicated internal server running antiviruses and only getting the software form that server This is true particularly for members of groups who visit FHW facilities as part of an agreement between. We made it clear to all during the preparatory phase that participation is not a requirement and has no effect.
P4
Participants feel pressured to participate in the experiment
low
high
Preparatory phase
avoid
9.2.2. Risks for the experiment

For the reasons listed above, the current instance of the risk register for the experiment is as follows:
Table 2. Risk register for the experiment.
ID E1
Description Required experiments cannot be run because 90% of thee visitors are children and adolescents and the agreed Ethical Oversight Measures state we will not be dealing with children Required EXPERIMEDIA components not available on time or not compatible with the FHW facilities Damaged mobile devices
Probability low
Impact high
Proximity Execution phase
Response accept
Comment Already run the first stage of the experiment successfully. Doing the experiment with only "pure" groups
E2
low
high
Experiment design
reduce
Reformulated as E5, E6, E7, E8
E3
low
medium Execution phase
reduce
We will run the experiment with as many working devices are available. The mobile devices only affect one part of the
43
EXPERIMEDIA
experiment, and therefore the feedback of participants that are not given a device is still relevant
E4
Weather does not permit the use of outdoors locations
low
low
Execution phase
reduce
We are planning to execute the experiments at a time that the weather is typically suitable Via close coordination with ICCS
E5
Social component not available on time or not compatible with the experimental facility Streaming component not available on time or not compatible with the experimental facility AR component not available on time or not compatible with the experimental facility Experiment monitoring component not available on time or not compatible with the experimental facility Delays in the implementation and integration of the facility make it impossible for the work to be completed on time
medium
medium Second phase of the experiment medium Execution phase
reduce
E6
low
accept
Already integrated successfully into the experimental facility Already integrated successfully into the experimental facility Already integrated successfully into the experimental facility
E7
low
low
Execution phase
accept
E8
low
high
Execution phase
reduce
E9
medium
high
Execution phase
reduce
Already integrated successfully into the experimental facility
44
EXPERIMEDIA
10. Evaluation
Using the questionnaires, interviews, on site observations and careful examination of the video recording we evaluated the impact of FIRE technologies, their suitability and identify parameters that affect them. The usage of a remote expert using live audio and video for real-time communication allowed for a exciting and professional setting for the visitors. The broadcast of the video feed from the Tholos was mandatory for the usage of remote experts but was also used to promote the program across the web using social media. We had many hits from external visitors who once the link was posted public tuned in to watch the show, thus creating a unique promotion event. The Facebook app and the ability of concurrent written communication of the visitors with an expert but also among themselves resulted in more questions being asked and answered. The whole walk-through got instantly more social and exciting allowing the visitors to acclimatize very quickly to the tour and become much more focused. Even the occasional whispers amongst friends diminished since they were using messages to communicate. The AR pre event provided the right amount of engagement and interest hook before an intense VR walk-through. The fun factor of such a exhibit is very high and besides letting the visitors interact with the exhibits provided the opportunity to effectively extend the reach and completeness of the program. While museums have long strived to be welcoming places as well as havens of learning, social media can turn them into virtual community centres. On Facebook or Twitter or almost any museum Web site, everyone has a voice, and a vote. Curators and online visitors can communicate, learning from one another. As visitors bring their hand-held devices to visits, the potential for interactivity only intensifies. However, there is a caveat. The new technology is stimulating but the programs have to make sure that people keep in a heads-up mode to make sure that they are also looking at the historical and art information presented.
10.1. Suitability of FIRE technologies

The experiments tried to enhance the visitor experience using three distinct future media internet technologies, social networking, high bandwidth live streaming and augmented reality. During the various experiments starting in 2011 as described in the reports D4.3.1, D4.3.2 we constantly revised the experiment plan according to the results we got from small scale experiments. The final experiment was a distillation of the best practices that could be incorporated into a traditional Tholos presentation and enhance it. These technologies not only worked seamlessly as expected but also were able to supplement each other forming a unified experience. As is evident from the questionnaires all visitors immediately recognized and appreciated their value.
10.2. Impact of FIRE technologies

The impact of a technology on visitors can only be measured by evaluating and analyzing the questionnaires. Analyzing results it is clear that the impact in our experiment were in two distinct fields.
EXPERIMEDIA
Visitor socializing: All technologies allowed the visitors to reach out to another and to the expert. Every addition made supported that goal and enabled fertile ground for social activities. The AR event allowed them to explore and interact as a group, the streaming and social network components to communicate inside the Tholos before and after the show with each other and the expert, and more importantly during the show without disturbing the formal presentation. Enhanced learning: Visiting a Dome virtual reality theatre is a unique experience by its self which enables learning in multiple levels. The AR event promoted a educational pre-event using visual cues combined with textural printed information. The visual cues intrigued the visitor to read the information and interact with the 3D representations. The social event allowed them to ask freely without hesitation during and after the show, the museum educators witnessed and increase in the amount of questions asked. The evaluation proves that by asking more questions the visitors were more focused on the presentation and were given a feeling of being able to interact with it. It was not uncommon to see visitors invent questions or post funny comments only to participate in the whole Facebook discussion with the expert.
10.3. Parameters that affected impact

The previous sections argued about the usefulness and impact of FIRE technologies in the enhancement of a traditional Tholos presentation. Nevertheless as stated earlier the final experiment resulted from experiences gathered during the previous ones. We established certain critical factors which could affect the success of using these tools for Dome shows and museums.
10.3.1. Ratio of Devices/visitor

To experience these technologies the museum relies on the visitor to bring his own equipment, meaning his mobile phone or tablet. If the ratio of devices per visitor is very low the social aspects and the enhanced learning is tremendously impacted. The visitors without mobile phones cannot participate in the AR and social activities. Inside the Tholos they can only interact with the expert before and after the show making oral questions.
10.3.2. Quality of Wi-Fi signal

Without a reliable and working internet connection these technologies are rendered useless. We specifically had to increase and install Wi-Fi spots so as to ensure the whole venue's coverage. The exception lies with the AR event which is autonomous in that respect.
10.3.3. AR user interface

Using the AR mobile app had to be very straight forward, in our experiment visitors had trouble starting up the app because we used the free Metaio framework which had a very long start-up time. We also relied on museum educators to show them how to use the app. All this should be avoided by using the commercial version. Although we provided only 4 AR interest points there was room for more since a 2-5 min pre-show event has to provide enough material to keep all the visitors busy. The average visitor count of each Tholos presentation is 100 visitors.
46
EXPERIMEDIA
10.3.4. Duration and order of AR event

Initially we tried to use the AR points after the show scattered around the museum in order to motivate visitor exploration. This proved not functional since many visitors could not find the interest points or had the time and energy after a VR walkthrough of 40 minutes. The AR event should be easily accessible and for a short period of time when combined with a traditional show.
10.3.5. Number of visitors

We run the last experiment with a number of 18 visitors and had 1 expert. The focus group discussion and through observation it resulted that the expert had difficulty in answering all these questions on time and could easily overlook questions or comments. Often the visitors had to wait before getting a response. As the Tholos is a VR system of 130 seats it is evident that a scale in visitors need a analogous scale in experts to have any real value.
10.3.6. Social app and web app UI

The user interface of the social app which is used by the visitors and web app of the expert are of utmost importance. The UI should be self explanatory, easy to use and should not force the visitor to look at it all the time. During all experiment runs we witnessed a lot of failures and problems in that regard. The UI of the mobile app was desktop oriented and required constant button presses and usage of menus. The answers were not refreshed automatically and the visitor for forced to press the 'Refresh button repeatedly. This resulted in visitors missing large parts of the show. The UI has to try keep people in a heads-up mode to make sure that they are also looking at the historical and art information presented. Therefore a simple design is needed which is easy to learn and enables the visitor to quickly find the interest point he want to make a question about. It should also feature some form of feedback in the form vibration or a visual indication when a answer to his question arrives. Figure 20 shows that user that look at their phone cannot watch the show.
10.3.7. Latency versus Quality

The streaming part of the experiment was very easy to incorporate and setup. Nevertheless we experienced major latency issues which ranged up to 7 seconds. During the show and when interacting with the expert we could accept a temporal loss of quality both in picture and in some for also in audio but the latency made it difficult to conduct a live ask and answer section. Unfortunately the ECC metrics did not include a latency count, we could only establish this parameter by video reference and local participation in the event. Figure 21 shows the live ask and answer question of the expert with the visitors.
47
EXPERIMEDIA
Figure 20. Users using the mobile app during the show. Using the mobile app prevents watching the show.
Figure 21. The final ask and answer section of the presentation, in which the visitors can chat live with the expert, seeing his video stream on the Tholos dome.
48
EXPERIMEDIA
10.4. ECC Metric results

Although the ECC and the monitor clients functioned flawlessly the resulting metric data were tricky to interpret and analyze. The output of the ECC was two excel files with text and number information where one document indexed into the other. Figure 22 and 23 shows the Metric Data and Meta Data that was measured during the final experiment run. Not that a search and copy into both files has to be done in order to find a specific metric of interest.
Figure 22. The Metric Data that is output by the ECC
Figure 23. The Meta Data that us output by the ECC.
49
EXPERIMEDIA
For example by analyzing the stream bitrate of both the Tholos and the expert video from his face camera we can extract the following charts (Figure 24) which show the bitrate of the main Tholos stream. Although a low spike can be seen in the interview the expert did not mention anything about the quality. Therefore the visual quality streamed to the expert can vary without compromising the result.
Figure 24. A chart which show the variation in Bitrate for the main stream of the Tholos which was streamed to the internet.
The following figure (Figure 25) shows the bitrate form the face camera of the expert. This video feed was streamed onto the Tholos Dome, for the live question and answer section. Although the bitrate is low and has a lot of spikes the visitors did not complain about the image quality. They complained more about the latency which was evident in some cases, if we had a latency metric we could compare the two charts. We assume that the latency would follow the spikes of the main bitrate.
Figure 25. A chart which shows the variation in Bitrate for the stream coming from the experts face camera.
The ECC output for the SCC component listed all Facebook related data like Top Comments, Number of Likes, Top Questions and many more. Unfortunately it did not support UTF16
EXPERIMEDIA
encoding and all the Greek answers appeared in unreadable format. In combination with difficult procedure to analyze the data because of the fragmentation in two large files, it was easier and more productive to look directly at the Facebook event page and skim through the answers. Although this was feasible for the limited visitors of the experiment it will prove difficult for a larger visitor base. Therefore this needs improving. The following figure (Figure 26) shows part of the Facebook event page, note that all users with a cell phone posted at least once, and all pictures had related questions. The ratio of device/visitors was 50%, meaning that half of the users had mobile phones and therefore could experience the experiment in pairs of two.
Figure 26. A Facebook group view of some comments and answers posted during the experiment.
51
EXPERIMEDIA
Appendix A. Informed Consent
52
EXPERIMEDIA
Appendix B. Device Lease
53
EXPERIMEDIA
Appendix C.
Structured questionnaire
EXPERIMEDIA . , . 10 . 1 : ;
2 : ;
3 : ;
4 : ;
5 : ;
54
EXPERIMEDIA 6 : ;
7 : ;
8 : ;
9 : ;
10 : ;
11 : ;
12 : ;
13 : ;
14 : ;
55
EXPERIMEDIA
15 : ;
16 : ;
17 : ;
18 : ;
19 : ;
20 : ;
21 : ;
22 : 1 ;
23 : 1 ;
56
EXPERIMEDIA
24 : 1 ;
25 : ( ) ;
25 : ( ) ;
26 : ;
28 : ;
29 : ;
30 : - ;
31 : ;
32 : smartphone;
EXPERIMEDIA
33 : ;
34 : ;
35 : ; ( )
36 : ; ( )
37 : ; ;
38 : ;
39 : 1 ;
40 : 1 ;
58
EXPERIMEDIA
59

D4.3.3 FHW Experimental Results and Recommendations v1.0 PDF

Uploaded by

Document Information

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

D4.3.3 FHW Experimental Results and Recommendations v1.0 PDF

Uploaded by

Copyright:

Available Formats

D4.3.

3 Experimental Results and Recommendations

Efstathia Chatzi (FHW) Christopoulos Dimitrios (FHW)

Copyright FHW and other members of the EXPERIMEDIA consortium 2013

Experimental facility architecture ................................................................................................... 11

Experimental facility implementation............................................................................................ 17

Data analysis ...................................................................................................................................... 32 7.1. Demographic first analysis: .................................................................................................... 32

Copyright FHW and other members of the EXPERIMEDIA consortium 2013

ECC Metric results.................................................................................................................. 49

Copyright FHW and other members of the EXPERIMEDIA consortium 2013

Copyright FHW and other members of the EXPERIMEDIA consortium 2013

Copyright FHW and other members of the EXPERIMEDIA consortium 2013

Goals and success evaluation criteria

Goal 3: Measure impact of FIRE technologies (Moderate success)

Copyright FHW and other members of the EXPERIMEDIA consortium 2013

Experiences offered before EXPERIMEDIA

Copyright FHW and other members of the EXPERIMEDIA consortium 2013

Figure 1. Tholos operation without EXPERIMEDIA extensions.

The experience examined in the embedded experiment

4. Experimental facility architecture

The agents participating in this experiment are

Live streaming experiment during the show

Augmented reality experiment before the show

Copyright FHW and other members of the EXPERIMEDIA consortium 2013

Figure 3. Flow of information and component diagram for the experiment.

Copyright FHW and other members of the EXPERIMEDIA consortium 2013

The monitoring component

Copyright FHW and other members of the EXPERIMEDIA consortium 2013

5. Experimental facility implementation

Streaming experimental facility

Augmented reality experimental facility

Figure 5. The development environment for the mobile application.

Copyright FHW and other members of the EXPERIMEDIA consortium 2013

Figure 9. Example of markers used in the augmented reality application.

Copyright FHW and other members of the EXPERIMEDIA consortium 2013

Copyright FHW and other members of the EXPERIMEDIA consortium 2013

Figure 13. Another view of the educators during the experiment.

Social networks experimental facility

Figure 14. The visual GUI of the mobile SCC application

Copyright FHW and other members of the EXPERIMEDIA consortium 2013

Figure 16. The remote expert web application

Copyright FHW and other members of the EXPERIMEDIA consortium 2013

Monitoring experimental facility

Copyright FHW and other members of the EXPERIMEDIA consortium 2013

Before the beginning of the experiment

6.1.1. Deploy the facility

6.1.2. Familiarize ourselves

6.1.3. Inform staff and schedule accordingly

6.1.4. Prepare the informed consent forms

6.1.5. 6.1.5. Prepare the questionnaire.

6.2.1. Preparatory phase

6.2.2. Execution phase

6.2.3. Data acquisition phase

6.2.5. Focus groups

Demographic first analysis:

7.1.1. Statistical views of the data

Copyright FHW and other members of the EXPERIMEDIA consortium 2013

4,11 4,11 1,56

1,44 1,69 1,72

Copyright FHW and other members of the EXPERIMEDIA consortium 2013

8. Ethics, privacy, PIA

Minimum ethical principles

8.1.1. Doing good