System Architecture of a Mixed Reality Framework

Helmut Seibert∗, Patrick D¨ahne†

Department Visual Computing
Center for Computer Graphics (ZGDV)
Rundeturmstraße 10, 64283 Darmstadt, Germany
email: helmut.seibert@zgdv.de
www: www.zgdv.de

Department Virtual and Augmented Reality
Fraunhofer Institute for Computer Graphics
Fraunhoferstraße 5, 64283 Darmstadt, Germany
email: patrick.daehne@igd.fraunhofer.de
www: www.igd.fraunhofer.de

Abstract Keywords: Mixed Reality framework, Augmented
Reality, data-flow management, distributed systems

In this paper the software architecture of a frame-
work which simplifies the development of applications
1 Introduction
in the area of Virtual and Augmented Reality is pre-
sented. It is based on VRML/X3D to enable rendering
Creating Mixed Reality (MR) applications currently is
of audio-visual information. We extended our VRML
one of the most challenging tasks in the field of com-
rendering system by a device management system that
puter graphics. The developer has to deal with exotic
is based on the concept of a data-flow graph. The aim
hardware devices like head mounted displays (HMDs),
of the system is to create Mixed Reality (MR) applica-
GPS receivers, electronic compasses etc. In many
tions simply by plugging together small prefabricated
cases, especially when building mobile systems using
software components, instead of compiling monolithic HMDs, traditional IO devices like keyboards and mice
C++ applications. The flexibility and the advantages are not available. No widely adopted standards exist
of the presented framework are explained on the ba- in this area.
sis of an exemplary implementation of a classic Aug-
Our own experience with different projects has
mented Reality application and its extension to a col-
shown that most MR applications are closely tied to
laborative remote expert scenario.
the hardware they have been developed for. It is usu-
ally not a trivial task to switch from a video-see-trough
head mounted display to a system based on a web pad,
Digital Peer Publishing Licence or to exchange an electromagnetic tracking system by
Any party may pass on this Work by electronic a video-based tracking system. In most cases, chang-
means and make it available for download under ing the system configuration results in rewriting major
the terms and conditions of the current version parts of the MR application.
of the Digital Peer Publishing Licence (DPPL). Another problem is that hardware platforms for MR
The text of the licence may be accessed and applications are becoming more and more heteroge-
retrieved via Internet at neous. Mobile AR applications for example are not
http://www.dipp.nrw.de/. necessarily limited to the usual Wintel platform (MS
First presented at the International Conference Windows operating system on an Intel microproces-
on Computer Graphics Theory and Applications sor), instead, new hardware platforms like handheld
2006, extended and revised for JVRB PCs, handheld game consoles and mobile phones are

no.g. object changes. appropriate interac- proach is that applications are platform independent – tion methods for AR applications. Our system shares these advantages with Im- ware frameworks gained more and more attention. Studierstube is based on the concept of a distributed scene graph (Distributed 3 Concept of the MR Framework Open Inventor). functionality typically needed by AR applications. COTERIE is a set of packages written our system is based on the concept of a data flow in Modula-3 that allow to create “Distributed shared graph. i. Ex- dent. The major advantage of this ap- sition and orientation (tracking). ageTclAR.e. ImageTclAR [OTX03] is an AR framework based tions. and a data flow framework called OpenTracker [RS01] that handles different kinds of As already mentioned in the introduction. to handle hardware devices. i. callbacks on these objects to get notifications when an vices makes cross-platform development even worse. VRPN ing. In contrast to ImageTclAR.Journal of Virtual Reality and Broadcasting. TERIE [MF96]. The basic runtime system of the Tcl script allows us to separate the hardware-independent appli. create MR applications that can be implemented with- DWARF [BBK+ 01] is using a concept similar to our out much effort. Applications written in these C++ API provides abstract classes to access devices. which in our opinion better reflects the event- objects”. But recently. markup languages are completely platform indepen. Our 2 Related Work system is not a library that has to be linked into the application code. changing the occlusion of virtual objects by real objects. the development of AR soft- code. third-party software and different scene graphs. without having to recompile any computers. Our framework differs from these existing solutions in the way it is used to create MR applications. The main focus of our system is rapid prototyp. gler [CNBH+ 02] and DIVERSE [AK02]. they can be executed on any platform the ternal devices and applications can be accessed via runtime system has been ported to. CORBA. we developed a framework that on Tcl/Tk. this situation is not very satisfactory and between different processes on the network. and can be easily ported to miscella- own concept described in this paper. transparent access to VR devices. Another well-known software-framework is Studierstube [SFH+ 02]. Changing the hard. ISSN 1860-2037 . urn:nbn:de:0009-6-7774. and the de- hardware configuration can be easily done by modify- velopment of powerful and yet lightweight wearable ing simple text files.e. AR applications neous hardware configurations. graphical problems the applications run on all platforms the runtime sys- like the correct illumination of virtual objects and the tem has been ported to. Instead of creating large. instead it is a runtime system that al- The main research focus in the area of Augmented lows to interpret MR applications written in a set of Reality has been the determination of the user’s po- markup languages. pend on the hardware configuration. Our framework access to video cameras and tracking devices. monolithic C++ applications that have to be recom- Tinmith [PT03] is using an object store based on piled each time the hardware configuration changes. 7 emerging.e. Furthermore. CORBA is used for all network communi. which also allows to develop applications An early example of a software framework is CO- in a scripting language. Objects are automatically distributed Obviously. scripting language. i. The application logic are build by using services that form a distributed data should be clearly separated from the code necessary flow graph. cation. Unix file system semantics. allows to develop MR applications by using VRML MORGAN [OHFB04] is a component-based frame- and a special markup language used to describe the work for dynamic multi-user AR and VR projects. ware configuration can be simply done by modifying Other software frameworks that focus on the text files without the need to recompile any application development of VR applications are VR Jug- code. For this reason. hinders the introduction of commercial MR applica. data objects that are transparently shared driven nature of MR applications than a procedural by processes on different machines in the network. Volume 3(2006). A hardware configuration. our aim is to trackers. Applications can register figurations and operating systems on these kinds of de. language has been extended by modules that provide cation logic from the parts of the software that de. it is meant to be a research system that allows [THS+ 01] and IDEAL [FR00] are device management to rapidly develop small applications that serve as a systems that concentrate on the flexible and network- test bed for new algorithms. The broad range of different hardware con. e.

Volume 3(2006). Each node of the scene 1. the scene graph is sup- plemented by a data flow graph that contains the appli. We have live video and audio streams connected to. 2. But only few provide means to process the data for the basic. application-independent set of standard values. the developer has to derive a application to access devices by using a set of abstract new class from the abstract node base class. tracking systems or user interface to the framework. our framework is based on VRML. and nodes that consume data streams (output aim is to create some kind of a runtime system that is devices). we realised that at least graph in more detail. reusable software ing of data streams resulted in the concept of a data components that are not hard-wired. pletely device-independent manner. 7 we prefer to have a set of small. When it is not possible to implement the that transform coordinate systems. for example “Locator” for tracking sys. It is 4. Like in VRML. i. or to calcu. Nodes that transform data. but can be easily flow graph. outslots device. we had to extend our node that allows to integrate subgraphs stored in VRML framework by a device management system. In this graph. for VR applications the VRML standard to a large de- gree fulfils our needs.e. the set of ing systems that take video frames and transform VRML nodes can be extended by using “Script” nodes them to position and orientation values. work transparent. for VR appli. or they can be loaded systems into another coordinate system. be linked into the application executable (mostly used tems. Most existing device management systems allow the To create a new node.e. Nodes can device classes. data streams on a hard disk. no.g. ISSN 1860-2037 . filter and orientation values from tracking systems. These are usually de- graph is a small state machine that is able to receive vice drivers of input devices. Device driver nodes for input management system. nodes that transform data adapted to the specific system configuration. i. etc. change its state. Nodes that consume data. ues when they are notified that new data is available. This designed for Internet applications. configuration files. and the “Inline” live video streams. streams of position ily need to have their own thread. In fact. e. Nodes that produce data. and to send events depending data streams stored on a hard disk. and inslots are connected by “Routes”. as plugins during runtime (application-specific nodes). we have nodes that produce reconfigured by using configuration files that can be data streams (input devices). For example. It does The “Network” node that makes the system net- not provide any means to integrate devices and me. devices usually always need to have an own thread. the “Web” node that provides a dia used for MR applications.Journal of Virtual Reality and Broadcasting. The follow- When analysing existing standards and techniques ing sections describe the components of the data flow in the computer graphics area. vice drivers of output devices. nodes that store Because VRML perfectly fits to our requirements.and system-independent markup language. Our demand for extensive preprocess. the developer of a new node is whether the node uses When we planned the architecture of our device its own thread or not. All other types of nodes do not necessar- from cameras and microphones. The final streams. VRML is a markup language 3. But VRML has its weaknesses when used for MR applications. or video track- application logic by using standard nodes. Therefore. Nodes that do not deal with data in any way. on the state transition. late the position and orientation of the user by using An important design decision that has to be made by video images.1 Nodes that allows to describe the virtual scene in a com- Nodes are the core component of the data flow graph. timers etc. urn:nbn:de:0009-6-7774. sounds strange at a first glance. Besides the clas- There are four types of nodes: sical scene graph. Examples are nodes cation logic. These are usually de- Java. currently already has three nodes of this kind: dard IO devices like mice and keyboards. but our system cations running on a desktop PC equipped with stan. VRML also allows to specify the be- haviour of the virtual objects. whose behaviour can be implemented in JavaScript or 3. we realised that almost all de. Nodes send data values via “Outslots” and able to load and execute MR applications written in a receive values via “Inslots”. as well nodes that transform position values into another co- as button-press and -release events from game pads ordinate system usually directly calculate the new val- or keyboards. vices used in MR applications produce or consume because they have to listen to the port the device is data streams. to transform position values of tracking nodes provided by the system). nodes that replay events.

smart pointers are used to transfer large amounts of data. e. “Button #1” for the first button of a be created between nodes on different machines. the When the Network node starts operation. This allows application de- corresponds to the number of buttons. it auto- application developer simply has to create a so-called matically connects to all other Network nodes on the “Route” that maps the label of an outslot to the label of network.4 Namespaces just requires to copy a pointer.e. i. nodes available on the network without the need of urn:nbn:de:0009-6-7774.e. Volume 3(2006). hide im- plementation details. “Joystick 1” and “Joystick introduce network transparency into the device man- 2” for two nodes that operate joysticks attached to the agement system. tected by the joystick driver. Of course. velopers to structure their data flow graphs.g.Figure 1: Network transparency by creating proxy tomatically transfers copies of these data values to all nodes inslots it is currently connected to. Both outslots and inslots are typed. On the other hand. the outslot au. Node D you have to specify what kind of data can be send to or received from the slot when you create it. Graph 2 change data values.e. the number of float The developer can export any inslot and outslot from outslots corresponds to the number of axes provided the graph contained in the Namespace to the graph that by the joystick.g.Journal of Virtual Reality and Broadcasting. When a node writes data values into an outslot. They contain a com- ber of outslots depends on the joystick hardware de- plete data flow graph consisting of Nodes and Routes. filter nodes that have Node B to perform complex calculations. i. We are using a technique called “Multicast an inslot. called “Network”. and create macros that provide functionality to other developers. and inslots are used to receive data from Proxy B other nodes. Nodes of the graph communicate via system. To connect an outslot to an inslot. each Our system does not support network transparency slot in the system can be identified by a unique label directly. An interesting feature is the possibility to create For example.2 Outslots and Inslots Outslots and inslots are the means used by nodes to ex.g. Network 3. Each outslot and inslot of a node has a unique edges.5 Network Transparency When creating nodes of the data flow graph. like video trackers. no. Node C usually get their own threads. Please note that the actual num- data flow graphs just like Nodes. the ap- plication developer has to specify a unique name for The concept of a data flow graph makes it simple to each of these nodes. only slots sending and receiving DNS” [MDN] to automatically discover all Network the same data type can be connected by routes. and the number of boolean outslots contains the Namespace. joystick or “X-Axis” for the x-axis. it is made available by a special node consisting of the node name and the slot name. 7 This means that they are in fact driven by the threads Graph 1 of the device driver nodes that provided the incoming Node A data values. As a result. For efficiency rea- sons. and a boolean “Namespaces”. Instead.3 Routes 3. e. the Joystick node consists of two float hierarchies of data flow graphs by using so-called outslots for the two axes of the joystick. i. second joystick. Namespaces can be integrated into outslot for the button. ISSN 1860-2037 . copying data values into inslots usually 3. 3. We did this to achieve a clear sep- “Joystick 1/Button #1” for the first button of the first aration between the device management and the net- joystick or “Joystick 2/X-Axis” for the x-axis of the work code. e. Outslots are used to send data to Proxy A other nodes. so the most elegant solution is to allow edges to label as well.

this approach is not recommended. The configuration is loaded when starting the ap. Furthermore. AR system. which camera to use and plication. figuration file. 7 any configuration or central servers.Journal of Virtual Reality and Broadcasting. current state of the system. Figure 1 for example shows two graphs into one scene graph. as well as a short de- the integrated user interface. Instead. Instead. In the upper left is an overview of all nodes that face also allows to save the current status into a con. Instead. scription (upper right of figure 2). 3. The user simply loads the configuration by using currently available in the system. The figuration file. developer gets status information and error messages. when transferring to operate user interfaces that consist of windows. They can be used in the second graph just the same way as local nodes. but- video frames. sliders etc. Nodes preprocessor statement allows to integrate several A and B of the first graph are available as proxy nodes pieces of source code into one file. It is also possible to get a list of all node types 1. urn:nbn:de:0009-6-7774. a corresponding codec has to be implemented debugging directly on the machine the application is that performs this serialisation. The interface allows to inspect the editors and other free tools available. For each data that it is quite often impossible to do development and type.6 Configuration File Our solution is a small web server that provides The structure of the data flow graph is usually not hard a user interface build from HTML pages. no. exactly the same way as the VRML “In- nodes can be used to create edges between nodes on line” node allows to integrate several VRML sub- different machines. the Network nodes create local proxy nodes for tegrate configuration files into other configuration all nodes available on remote machines. in the second graph. to add and to remove nodes Even though it is possible to create these XML files and routes from the graph. he gets a page where he 2. This user inter. tion is running on. The web server solution allows cation again. plication from other machines in the network. these applications usu- ally run in full-screen mode which makes it difficult to watch any debug output in console windows. to change parameters. Such wearable systems often do not have Most codecs simply write the data values into the byte conventional input devices like keyboards and mice. When the developer decides to create a new node. e. These proxy files. but more sophisticated codecs are possible that This means that the user of such a system is not able compress the transferred data. we use a configu. terface of the device management system to create a Figure 2 shows some screen shots of the web inter- data flow graph that fulfils his needs. e. established standard is connected via network to the machine the applica- for storing information. face. to the data flow graph. but the application programmer has to implement and For example. After connect. tons. This web coded into the application. There is a special “Inline” node that allows to in- ing.7 User Interface sent between the proxy nodes and local nodes of the second scene graph are automatically transferred via Our device management system has an integrated the network. us to control the device management system from any The format of the configuration file is XML due to device that has a simple web browser installed and that the fact that this is a well-known. server is not part of the device management system. but any events 3. can specify parameters. stream. During our connection needs to be transformed into a system. currently exist in the device management system. Volume 3(2006). ISSN 1860-2037 .g. and he can add a configuration file: nodes by selecting the type of node from a drop-down menu. 3. and by hand. or the C “include” data flow graphs running on different machines. consider the development of a mobile register codecs for all application-specific data types. There are numerous parsers. work on several MR projects it soon became obvious independent byte stream (serialisation). Graphical User Interface (GUI) that allows to mod- Each data value that gets transferred via a network ify the data flow graph during runtime. Such codecs already running on.g. we need means to control our ap- exist for the default data types provided by the system. There are three ways to restore a saved graph from He can remove nodes from the list. ration file to store the whole data flow graph on hard instead it is implemented as a node that can be added disk and to restore the graph when starting the appli. to save the current state to or to restore it from a con- the application developer uses the integrated user in.

no. ISSN 1860-2037 . Volume 3(2006).Journal of Virtual Reality and Broadcasting. 7 Figure 2: Screen shots of the web interface that allows to control the device management system. urn:nbn:de:0009-6-7774.

ple geometries. is comparable to the overhead of conventional frame- ternal device that has a web browser outweighs these works. Furthermore. The slot gets the it is possible to use a live video stream or alternatively name specified in the “label” field.1 Authoring xSensor { Development of applications using the proposed SFString [] label "" framework can be done by creating configuration files SFBool [] out FALSE x [in. . We are looking for better solutions that allow structure of the system. . which is unhandy for managing rendering – instead. for the concrete VRML field types SFBool. . But the by tracking and rendering. tem. we defined a sensor node. e.Journal of Virtual Reality and Broadcasting.out] value by hand. the overhead benefit of being able to control the system by any ex. An authoring framework which offers existing Nodes For example. we do not use scripting and markup languages to Obviously. It is handle time-consuming tasks like video tracking and completely text-based. Volume 3(2006). The VRML scene to load previously taken camera images of the scene sends data to outslots or receives values from inslots in order to place the augmentations. These allows the developer to choose the appropriate devices sensors are defined as follows (“x” is just a placeholder for the desired application.g. For each field data type prefabricated pool of nodes interfacing various devices available in VRML. The authoring application manip- } ulates the data flow graph and the scene graph of our Mixed Reality framework using its external authoring For each sensor in the VRML file we create a corre. see figure 3. no. the sensor for boolean events is defined and creates the required Routes based on prepared con- as figurations is recommended. interface. Another problem is that the web interface by scripts. The communication between the modules only allows the get a snapshot of the current status of of course introduces some overhead – but this over- the system – it is not possible to get a notification when head is small compared to the calculation time needed some important change in the system occurs. they are only used to set up the a graph. or he can create guages often suffer from the prejudice that they are new routes by selecting outslots and inslots from drop. but SVG is natively only (C++) – only the interconnection between the software supported by the newest version of the Firefox web modules (nodes) and the application logic is controlled browser. pends on the value of the “out” field). ing are implemented in an efficient compiled language ing SVG instead of HTML. SFFloat. The augmenta- via the “value” field. MFBool. The developer can remove routes. inefficient. This is not generally true. ISSN 1860-2037 . . MFFloat. authoring is based on camera views of the real scene. there is a page that shows all routes that are currently defined (lower right of figure Applications written in scripting and markup lan- 2). In our sys- down menus. but even at that high level the creation of sim- } ple applications is error prone and becomes tedious. ): 6. . .out] value FALSE guides interactively. A VRML is straightforward. so that the current state is always visible to sponding outslot or inslot in the graph of the device the author during the whole authoring process. see [BDR04]. disadvantages by far. The whole processing urn:nbn:de:0009-6-7774. 7 which resolution and image format (see lower left of 5 Performance Considerations figure 2). As the management system (if it is an outslot or an inslot de. Video tracking and render- to present a graphical version of the graph. Furthermore. tions can be chosen from a content database or from For more information about the design rationales of a set of basic augmentation elements like text or sim- the sensor nodes. by us. 6 Applications 4 Integration in VRML/X3D The proposed architecture simplifies and speeds up the The integration of our device management system into development of new Mixed Reality applications. . The following applica- SFBoolSensor { tion example has been developed using a simple au- SFString [] label "" thoring framework which offers to create step-by-step SFBool [] out FALSE SFBool [in. the web interface has its limitations.

More sophisticated Mixed Real- ity applications require an enhanced authoring frame- work. one for the data flow graph. and information on how to nector. authoring will not be discussed in detail here.2 AR Application using marker-based tracking This section focuses at the development of an Aug- mented Reality application using a marker-based video tracking system based on the well known AR- ToolKit [KB99]. Figure 4: User with head mounted display. and one for the scene graph.g. Volume 3(2006). a standard USB video camera for the observation of the real scene and a Sony LDI-100 high resolution head mounted display for the observation of the augmented scene.Journal of Virtual Reality and Broadcasting. which Figure 5: Augmented view. Figure 3: Example authoring application for step-by- step guide applications. which incorporates e. and the result is visible in the users view. As the main focus of this paper is on the design of the data flow graph. see result of the authoring process is a set of configura. A common task in Augmented Re- ality applications is to enhance the view to a real scene with virtual objects containing specific informa- tion about real objects. nonlinear storytelling techniques like described in [SG06] to provide more opportunities in design and composition for the author and better interaction as well as more information to the user. 6. which can then be used with the mixed reality framework.figures 6 and 5. establish a defined cable connection. The to the correct connector of a network device. e. no. 6. urn:nbn:de:0009-6-7774. Three buttons are mounted on the hand-held display to enable simple user interactions like switching application modes or adjusting parame- ters.1 Hardware Setup The hardware components for the user are a laptop. allowing the user a plug a cable output window of the mixed reality framework. ISSN 1860-2037 . tion files.g. showing location of a con- are known from a database. where the augmented view on a network-switch de- vice explains the allocation of its connectors. We chose a technical scenario. see figure 4. 7 pipeline including the tracking tracking algorithms is Detailed instructions can be used to augment the driven with these images.2.

The system is implemented as a video-see through. The web interface allows a look inside the system and culated by the external tracking node.Journal of Virtual Reality and Broadcasting. The appli. The Tracking Node has to be ini- tialised with a number of pattern descriptions.3 Software Components and Data Flow terns to update the corresponding Transform nodes in To implement the mentioned AR application.2. Two outslots are created for each computer vision algorithms and optional filters that al. the nodes and routes are created.g. A sample VRML file video tracking system consists of a video source. see figure the rendering system is supplied through an additional 8 for a sample configuration file. sources of error can easily be scribed device management system is integrated. calibration file. parameters. frustum and the image segmentation threshold. we are its scene-graph.4 Video Node in the scene-graph.2 Rendering system starts. Volume 3(2006). to specify image size or image format to initialise the video node in a suitable way. by means of virtual objects to be in- cluded in the augmented view for the user is defined 6. so video images are drawn as back. Viewfrustum. The video node produces video data by interfacing the ground to the application window.2. The integrated web interface shows Rendering is done with the Avalon [BDR04] render. To toggle the visibility of the corre- able to select prepared modules. The detected or not is also provided. Other parameters are the URL of the camera Figure 6: Augmented view. One system. All marker transformations are use are described briefly in the following. if there is no can be received via VRML sensor nodes. the near and far values for the view mation about how to plug in a connector. if necessary. and the pattern size for each used marker. grouping of several markers. 7 6. to examine their state. environment. which allow to recognise trained patterns. The nodes we slot of type SFBool. Each frame marker. and another out- low e. When the application urn:nbn:de:0009-6-7774. the video node will specific extension we use is the Viewfrustum node fail to initialise resulting in a blank application screen. no. camera connected to the system. The de. The scene-graph camera and. which allows to use a projection matrix which is cal. all nodes and routes that have been created and allows ing framework which utilises VRML/X3D. Transform node and Switch of the video sequence is pushed through an SFImage node are updated according to corresponding sensor outslot as a single image. the is shown in figure 9. Optional parameters allow nodes. The files are retrieved and the module creates the outslots for each pattern. calculated for the camera model defined by the cam- cation can be implemented by setting up a data flow era calibration file and additional near and far plane graph holding the corresponding nodes and routes con.2. showing additional infor. For example. converting compressed video includes a Transform node and additional child nodes formats to a suitable uncompressed format for the vi- representing the corresponding virtual object for each sion algorithms and the display system. Video frames are re- ceived through an SFImage inslot and processed with computer vision algorithms. The rendering system needs to be supplied with the transformation of the recognised pat- 6. and the system is ready for use. data determined and eliminated. The corresponding projection matrix for necting corresponding inslots and outslots. 6. ISSN 1860-2037 .5 Video Tracking Node The video tracking node provides the tracking algo- rithm to compute the transformation of each recog- nised marker in the video image. one outslot of type SFMatrix. pattern. as illustrated in figure 7. This is done by supplying a parameter string containing the pattern name and the URL of the corresponding pat- tern definition files. The augmented shows the existing nodes and their current state. represented as nodes sponding object. the information whether it has been which can be inserted into our data flow graph.2.

ISSN 1860-2037 .dat"/> ROUTE imageSensor."/> DEF modelSelectSensor BoolSensor { <ExternalRoute internal="*" external="{SlotLabel}"/> label "Model select" </Node> description "Model select Sensor" <Node type="Network" label="Network"> } <Parameter name="ID" value="3002"/> ROUTE modelSensor. <?xml version="1.value_changed TO model_trans.0 <Route from="ARToolkit/M1" to="MarkerGroup/M1"/> } <Route from="ARToolkit/M1 select" to="MarkerGroup/M1 select"/> DEF projectionSensor SFMatrixSensor { <Route from="ARToolkit/M2" to="MarkerGroup/M2"/> label "Projection" <Route from="ARToolkit/M2 select" to="MarkerGroup/M2 select"/> description "Projection Sensor" <Route from="MarkerGroup/Group" to="Model transform"/> } <Route from="MarkerGroup/Group select" to="Model select"/> ROUTE projectionSensor.value_changed TO model_trans.7"/> DEF inlineNode Inline { <Parameter name="markers" value="m1.http://pc978/markers/m1.5 0. Figure 9: VRML file for the rendering system.set_matrix <ExternalRoute internal="*" external="{SlotLabel}"/> ROUTE modelSelectSensor.wrl" m2.0 0 0 1.patt.Journal of Virtual Reality and Broadcasting. urn:nbn:de:0009-6-7774.0 0 0. 7 Video VRML Scene SFImage Frames SFImage Frames SFImage Projection SFImage Model transform SFImage Model select ARToolkit SFImage Frames in SFImage Frames out MarkerGroup SFMatrix Projection SFMatrix M1 SFMatrix M1 SFMatrix Group SFBool M1 select SFBool M1 select SFBool Group select SFMatrix M2 SFMatrix M2 SFBool M2 select SFBool M2 select Figure 7: Data flow graph for example application.15. no.122 -404.value_changed TO inlineNode.set_projection <Node type="Video" label="Video"> DEF imageBG ImageBackground { <Parameter name="Width" value="640"/> texture DEF image PixelTexture {} <Parameter name="Height" value="480"/> } <Parameter name="Format" value="RGB24"/> DEF imageSensor SFImageSensor { <ExternalRoute internal="*" external="{NamespaceLabel}/{SlotLabel}"/> label "Frames" </Node> description "Image Sensor" <Node type="ARToolkit" label="ARToolkit"> } <Parameter name="camera_param" value="http://pc978/philips. url"http://pc990/augmentation.value_changed TO image. Volume 3(2006).15.patt. } m2.set_url Figure 8: XML configuration file for the device management.set_visible </Node> DEF urlSensor SFStringSensor { <Node type="Web" label="Web"> label "augmentation_url" </Node> description "augmentation url Sensor" </HID> } ROUTE urlSensor.0 <Route from="ARToolkit/Projection" to="Projection"/> zFar 5000.0 0 0 1.set_image <Parameter name="far" value="5000"/> <Parameter name="near"value="50"/> DEF model_trans MatrixTransform { <Parameter name="threshold" value="100"/> children [ <Parameter name="quality" value="0./> } <ExternalRoute internal="*" external="{NamespaceLabel}/{SlotLabel}"/> ] </Node> } <Node type="Group" label="MarkerGroup"> DEF modelSensor SFMatrixSensor { <Parameter name="name" value="group"/> label "Model transform" <Parameter name="mode" value="linear"/> description "Model transform Sensor" <Parameter name="markers" value="m1.0 utf8 <Route from="Video/Frames" to="ARToolkit/Frames in"/> DEF matrixCam Viewfrustum{ <Route from="ARToolkit/Frames out" to="Frames"/> zNear 50.0"?> <HID versionMajor="1" versionMinor="0"> #VRML V2.http://pc978/markers/m2.value_changed TO matrixCam.

Thomas Re- sound hardware. Video image and tracking information are SFImage outslot at the same time the marker informa. ISSN 0730-0301. and provides the Graphics (TOG). MacOS X. ber of bits to sample. Further- more a bidirectional speech communication is neces. Proceedings of ACM SIGGRAPH 2002. to move a pointer and place additional virtual objects and instructions into the view of the worker. 7 outslot of type SFMatrix.References sary to extend the information interchange. Arsenault and J.2. no. Mouse inter- actions are grabbed by the mouse node and provided on an outslot of type SFVec2f. icher. vol. at the expert system and an inslot in the rendering sys- plying appropriate offset transformations to the pattern tem of the user.6 Group Node tem has an overlay which is shifted on the screen ac- Tracking with single markers does often not satisfy the cording to the mouse movements of the remote expert. The 387. remote expert by changing the URL field of an Inline The use of redundant markers is possible if the dis. 3. The DI- nection. The num. Interactions VERSE Toolkit: A Toolkit for Distrib- are done by mouse and keyboard.3 Extending the Application experiences we gain from the practical use of the pro- posed system are incorporated into further develop- The application above has been enhanced to allow a ment steps of the system. 2002. In this paper we presented the framework we use for bined by applying a linear combination of transforma. design and implementation of new expert at a different location. are optional pa. It is cur- tions as introduced in [Ale02]. counterpart is the sound-play node.Journal of Virtual Reality and Broadcasting. Our current research focus simple communication between user and an “remote” is the identification. because some markers get oc. which can be provided for initialisation. Kelso. The inslots and corresponding offset transformations have to be pro. to avoid displaced augmentations tablish routes between inslots and outslots on different the processed video frames are forwarded through an machines. A group node provides inslots for each sponding string is transferred via an SFString outslot pattern and calculates the group transformation by ap. The expert is supplied nodes that further ease the development of MR appli- with the camera images of the distant user. 21. tion is passed though the corresponding outslots. The 6. which is mapping each marker trans- formation to a group transformation. VR 2002. transformations. The vision algorithms The Network transparency of the system allows to es- are time-consuming. Two nodes at each system enable sound recording and [Ale02] Marc Alexa. rently available on several operating systems (Win- tion is then provided at the outslot of the group-node. pp. ACM Transactions on tiser of the installed sound hardware. The geometry of the overlay can be modified by the cluded or unrecognisable when the camera is moved. the corre- ing runtime. The sound-record node accesses the digi. Stefan Riss. The group transforma. Asa MacWilliams. needs of the application. Christian Sandor. node in the scene graph of the AR-System. see figure 8 for example. dows. needs to be specified by the remote expert. Gudrun the sample values to the digital/analog converter of the Klinker. a headset and network connection. and is able cations. the development of our MR applications. The URL placement in between is known and remains fixed dur. 380– rameters. which has the same scene-graph loaded. Linear combination of playback. Volume 3(2006). The group transformations for each pattern are then com. 7 CONCLUSION vided as parameters. Special Issue: PCM data on an outslot of type SFSample. Linux. IRIX and SunOS) and as a Java package (written in pure Java.[AK02] L. The user of the uted Simulations and Peripheral Device AR system has the equipment mentioned above plus Services. sound hardware and a headset. 2002. which receives the samples using an inslot of type SFSample and passes [BBK+ 01] Martin Bauer. ISSN 1860-2037 . no. routed to the rendering system of the remote expert. sample rate etc. The user of the AR sys- 6. and urn:nbn:de:0009-6-7774. Bernd Br¨ugge. The expert requires a workstation with network con. therefore running on all systems providing a Java virtual machine). E. transformations.

Inte- ceedings of the ACM Symposium on gration of Multidimensional Interaction Virtual Reality software and technology Devices in Real-Time Computer Graph- VRST’01. 7 Martin Wagner. ware and technology UIST’96. Computer Graphics pp. 2nd IEEE and and Virtual Environments 11 (2002). T. 83 – 94. 2004. and A. pp. Network- ISBN 0-89791-798-7. Presence: Teleoperators ity Conferencing System. A. Hesina. pp. 2003. ISBN 1-58113-907-1. MORGAN Framework: Enabling Dy- namic Multi-User AR and VR Projects. Interactive Storytelling. and W.An Open International Symposium on Mixed and Source Platform for Virtual Reality Ap. 166–169. An object-oriented software architecture [CNBH+ 02] Carolina Cruz-Neira. An Open Software Architecture for Virtual Reality Interactiond. Rainer Malkewitz. VR Juggler . Forum. M. E-learning and Games (Edutainment) Language-level support for exploratory (2006). Szalavari. VRPN: A Device-Independent. and Fan 2001. 2002. ACM symposium on user interface soft.org/. IETF draft. based augmented reality framework. Augmented Reality (ISMAR’03). Juliano. The Studierstube Augmented tion for a Video-Based Augmented Real- Reality Project. Seeger. 313–319. 71–78. Thorsten Virtual Reality VRST’01. 2001. he 1-58113-427-4. C. pp. Proceedings of the 9th annual [THS+ 01] R.Journal of Virtual Reality and Broadcasting. Story Pacing in http://www. Transparent VR Peripheral System. pp 419–428. Taylor. [MDN] Multicast DNS. ISBN 0-7695-1375-1. stieg. Xiao. ACM International Workshop on Aug- no.multicastdns. 55–61. G. 33–54. ics Applications. 2001. VRST’04. Hudson. for 3D mixed reality applications. pp. and ceedings of the second IEEE and ACM Kevin Meinert. H. [SG06] Felicitas Becker Stefan G¨obel. Design of a component. Arthur Tang. Zs. Allen Bierbaum. Proceedings of the International Conference on [MF96] Blair MacIntyre and Steven Feiner. J. pp. 1996. tem for Augmented Reality. Proceedings of 40th AIAA ISBN 0-7695-2006-5. Pro- [FR00] Torsten Fr¨ohlich and Marcus Roth. 1999. M. Proceedings cus Roth. mented Reality. [SFH+ 02] D. Pro- Patrick Hartling. vol. 1. Weber. ISSN 1467-8659. Volume 3(2006). L. Utilizing X3D for Immersive of the International Workshop on Soft- Environments. 247. IEEE and ACM International Sympo- sium on Augmented Reality (ISAR’01). [OTX03] Charles Owen. Pro- ceedings of the ACM symposium on [OHFB04] Jan Ohlenburg.[RS01] Gerhard Reitmayr and Dieter Schmal- hibit ’02. A. M. 45–54. Encarnac¸a˜ o. and Wolfgang Broll. Aerospace Sciences Meeting and Ex. Christoper Just. [PT03] Wayne Piekarski and Bruce H. pp. pp. 19. technology 2004 (WEB3D’04). plications. Iris Herbst. 47–54. Gervautz. ISBN 1-58113-427-4. and Mar. Schmalstieg. [KB99] Hirokazu Kato and Mark Billinghurst. 2000. ISBN 1-58113-845-8. ImageTclAR: A Blended Script and Compiled Code Development Sys- [BDR04] Johannes Behr. Helser. ISBN Froehlich. programming of distributed virtual envi- ronments. T. Proceedings of the ACM Symposium on Virtual Reality software and technology urn:nbn:de:0009-6-7774. Patrick D¨ahne. ISSN 1860-2037 . 2004. p. 85–94. ISSN 1054-7460. Pur- Marker Tracking and HMD Calibra- gathofer. Fuhrmann. no. 2003. Thomas. Proceedings of the ninth ware Technology for Augmented Real- international conference on 3D Web ity Systems.

February 2007. 3(2006). 7. urn:nbn:de:0009-6-7774. Volume 3(2006). no. no. Patrick D¨ahne. Journal of Virtual Reality and Broadcasting. urn:nbn:de:0009-6-7774.Journal of Virtual Reality and Broadcasting. System Architecture of a Mixed Reality Framework . ISSN 1860-2037 . 7 Citation Helmut Seibert. ISSN 1860-2037.