2nd IFAC Symposium on Telematics Applications

Politehnica University, Timisoara, Romania

October 5-8, 2010

A User Interface Framework for the

Control of Component-Based Multi-Robot
Systems ?
Antti Hölttä ∗∗ Jari Saarinen ∗∗∗ Antti Maula ∗∗∗
Aarne Halme ∗∗∗

Aalto University School of Science and Technology Department of

Automation and Systems Technology PO Box 15500 FI-00076 Aalto
∗∗
antti.holtta@gmail.com
∗∗∗
firstname.lastname@tkk.fi

Abstract: This paper presents a user interface study for highly distributed system. The user
interface is developed to control a multi-robot system, which is based on “component-based”
architecture using hardware abstraction. This paper present the general principles of the user
interface and provides an example of Graphical User Interface for controlling a generic fleet of
machines.

Keywords: Distributed systems, Mobile robots, User Interfaces

1. INTRODUCTION an independent process. The current trend is also to

Multi-machine system is in essence a complex distributed
computing system. One machine is usually by itself a
distributed system consisting of one or more buses, sensors
and actuators spread around the machine. Also, the pro-
cessing can take place in one or more processors running
multiple processes. A multi-machine system consists of
multiple mobile units which communicate with each other
as well as with system level components and users.
In the area of distributed computing, the concept of mid-
dleware has existed already for decades. The purpose of
a communication middleware is to hide the differences
in the system (network, operating systems, computing
environments) and make the system transparent regardless
where the distributed applications are running. The need
for similar mechanisms has also been identified in the
area of robotics. Currently there are number of software
projects (Kramer and Scheutz (2007); Mohamed et al.
(2008, 2009)) initiated by research organizations that aim
to provide general purpose robot control software. The
generalization is based on some communication middle-
ware solution combined with hardware abstraction. The
hardware abstraction aims to provide a description of a
capability of a robot in generic way, so that the same
interface could be used regardless of physical appearance
of the robot.
At the same time, more and more projects move towards
component based structure of the control software. The
idea behind component based structuring is that each
component defines a small, well defined, input-output
description. By grouping components together one can
achieve larger functionalities. A component is usually
? This work has been supported by Finnish Academy of Sciences
in context of Center of Excellence in Generic Intelligent Machines
research
incorporate ideas from service oriented architectures to
component models. Service based systems provide higher
level communication patterns for developers, and for the
systems, such as service description, service discovery,
and of course publish/subscribe methods. The motivation
behind service based communication is that each unit
would be able to describe its abilities through a standard
set of services which can be dynamically used by the
system.
The result is that a component based multi-robot system
has significant amount of processes that are distributed
across the network. The robots, which are distributed spa-
tially, are running a control software, which can partially
be processed on-board and partially somewhere else. Yet,
the user should be able to perceive the relevant parts of
the system as one application. This is a challenge from in-
formation technology point of view, but also from usability
point of view.
This paper introduces a user interface concept for control-
ling a generic multi-machine system. The system is based
on a distributed, loosely component based architecture
which implements service mechanisms. The ultimate goal
is to create a user interface framework, which could be used
to control any fleet of machines in specific applications.
Not only this requires that the system is generic, but the
user interface should be constructed in a generic fashion,
while supporting the usage of it in specific way.
The rest of the paper is organized as follows; Chapter 2
presents a short review to related work, Chapter 3 intro-
duces the overall system architecture, which is to be con-
trolled, Chapter 4 introduces the user interface framework,
its design criteria and implementations, Chapter 5 gives a
short description of applications the framework has been
used so far and Chapter 6 provides conclusions, lessons
learned and future work.

978-3-902661-84-5/10/$20.00 © 2010 IFAC 133 10.3182/20101005-4-RO-2018.00042

TA 2010
Timisoara, Romania, Oct 5-8, 2010
2. RELATED WORK
There exists numerous frameworks for robotic applica-
tions (Kramer and Scheutz (2007); Mohamed et al. (2008,
2009)). Usually the emphasis of research is on the un-
derlying technical features such as communication and
interfacing (Brooks et al. (2007)). The common motivation
of these frameworks is to support the application develop-
ment process by providing reusable modules for common
algorithms. More often than not, the user interfaces and
visualization of system are considered application specific
issues and not considered as part of the generic framework
development (this is also evident from the lack of publi-
cations from the area). Most often the generic graphical
utilities are developed for the development and debugging
purposes.
One common feature found from the frameworks is the
visualization of the state of the nodes in the network
(Chaimowicz et al. (2003); Calisi et al. (2008)). This means
that the active components are visualized, and often some
actions to control the components (Chaimowicz et al.
(2003)). ROCI browser also enables the remote code injec-
tion and starting the components remotely. Additionally,
Orca (Brooks et al. (2007)) and RTM-Aist (Ando et al.
(2005)) provide graphical tools for dynamically defining
connections between system components in a Simulink-like
manner.
The generic frameworks for controlling the frameworks
seems to be sparse. vIsage (Lipski et al. (2009)) is a debug
and visualization framework for distributed systems. It is
based on its own components for communication, and even
used for robotic applications. It has not been integrated to
any robotic frameworks, and does not provide controls over
the system. OpenRDK’s (Calisi et al. (2008)) RConsole
can be used for controlling robots with specialized viewers
integrated to the program. Furthermore, in all cases there
are graphical user interfaces, but as far as authors are
aware these are developed case by case.
The main contribution of this paper is to address the
need and possibilities for developing generic user interface
frameworks, which are integrated into the robotic devel-
opment environments.
3. INTRODUCTION TO OVERALL SYSTEM
ARCHITECTURE
The system architecture is based on an in-house developed
software called MaCI (Machine Control Interface). MaCI
is a component based robot control library. MaCI com-
munication is based on GIMnet (Saarinen et al. (2007))
which provides basic messaging middleware. GIMnet is
designed for remote control of machines over the inter-
net. It implements network transparency by forming a
virtual private network, which is maintained by separate
processes running in the network. The communication is
based on TCP/IP and requires new connections to sin-
gle direction only, allowing establishment of connections
even through firewalls. Additionally, GIMnet provides API
layer (GIMI), which provides all connection related func-
tions, as well as communication patterns for push/pull and
publish/subscribe implementations.
134
MaCI is similar to other related projects such as Orca
(Brooks et al. (2007)), Player/Stage (Gerkey et al. (2003))
and many others, except for its communication part.
The most important part of the MaCI is the interface
definition, which describes datatypes and messages, and
provides API’s both for implementing the service (server
side) as well as taking service into a use (client side). Im-
plementing a service requires that the server functionality
is implemented by the application developer. Using the
service requires only accessing the server using client, that
is, for the specific service the client side is always the same
regardless of what is behind the server implementation.
This functionality, also known as hardware abstraction,
provides the starting point for the development of the
user interface for the system. Instead of visualizing the
application specific data, the user interface visualizes the
MaCI interfaces, thus providing a generic framework for
MaCI system visualization.
MaCI implements a service listing system, that enables
dynamic discovery of available services running in the
network. The services are identified by using a MaCI Ser-
vice Locator description (MaCISL). MaCISL is a naming
principle, which provides means to give hierarchy and iden-
tification for the components. MaCISL consists of three
parts: device group, interface name and instance name.
Interface name is a standard (within the system) and spec-
ifies the type of service being accessed. Instance names rep-
resent independent instances of same interface type (e.g.
separates left camera stream from right camera stream).
The device group allows grouping any number of devices
to form hierarchical structures. For example, MaCISL
’FutureWorksite.J2B2.MaCI Ranging.Laser’ refers to an
Ranging interface, which instance is ”Laser“, it belongs
to a machine called ”J2B2“ which is part of a Future
Worksite. Devices can register to any group despite their
physical location, allowing to form logical machines from
devices distributed on arbitrary number of computers. The
MaCI service discovery does this transparently, without
the need for an application to know about the physical
location of the services.
The MaCI allows the development of completely dis-
tributed systems, where all the functionalities are running
in separate processes, to more centralized systems, where
the machine only provides certain services for the system.
In any case, the user interface uses the functionalities
provided by the MaCI to discover the available services
within the system. The service structure is already de-
signed so that it gives a user an intuitive view of the
components in the system. Furthermore, MaCI interface
definition provides good starting point for developing the
user interface components, as the ”client-side“ API of the
interface remains identical.
4. USER INTERFACE FRAMEWORK FOR A
DISTRIBUTED SYSTEM
4.1 General design criteria for User Interface
In general, user interface should allow a user to utilize
the system without extensive knowledge of the technical
details of it. User interface has to make the system and
its distribution transparent to user. Transparency in this
TA 2010
Timisoara, Romania, Oct 5-8, 2010

case means that the user perceives system parts, such as

robots, as whole entities, not as the group of modules that
form them. Further, the user interface should minimize
user’s need to know exact configurations required for
specific tasks. A concrete requirement is also support for
development and debugging. Finally the user interface
should be generic, adaptable and expandable enough to
meet the needs of the future applications of the system.
For the user interface, the following requirements were
identified:
• Visualization of the system on a general level: Show
network modules and connections between them
• Control of the system on a general level: Instantiation,
configuration and monitoring of modules.
• Using the system for its specific purpose, such as
controlling robots in specific applications
• Expandability: enabling future applications
• Support the freedom in implementation (in research
environment, one should not limit the innovation by
default)
Visualization of the system consists of showing the mod-
ules of the network and connections between the modules.
Control over the system means that the user can start the
system, (robots, devices, servers etc.), configure the con-
nections between modules and monitor the module states
(work load, errors, debug output). In the case of MaCI and
GIMnet, expandability denotes the possibility to add new Fig. 1. Gimbo’s user interface: left side box shows the
robots to the system, new devices to robots and to use the network’s services and right side box lists available
robots for new applications without reimplementing the components.
existing user interfaces. When developing new features to use what ever development environment as long as the
the user interface, developers should be allowed to freely protocol is fullfilled). These functionalities are also given
choose the programming language and libraries. through C++ API and by providing example components
for the developers.
4.2 UI Framework The use of system’s applications is achieved with reusable
components that can be combined to more complex user
User interfaces has to provide for the user all relevant interfaces. Component model also enables expandability
information and controls over the system. Hence, the com- and grants developers the freedom to choose their tools.
plexity of the system directly affects the user interface.
Especially, when the system is very generic, a single user 4.3 Gimbo
interface that would be suited for multiple applications
becomes difficult to implement and often results in poor GIMnet browser (Gimbo) (Fig. 1) is a graphical user
usability and extendability. An alternative is to develop interface for visualizing the state of the GIMnet and for
specialized UIs for different applications. This approach its applications. The nodes and services of the network
still faces the same problem, but in a smaller scale: changes are displayed in a tree-like list. Gimbo keeps track of
in the application, such as new tasks or machines, may the networks state by periodically performing the service
require reimplementing the UI. As oppose to centralized discovery.
user interface design, our approach aims to develop de-
centralized user interface architecture. The structural com- Gimbo provides three views over services: by network
plexity of traditional approaches is avoided by having a nodes, service types and by MaCI naming convention. Due
component structure. The idea is that a user may execute to the maturation of the system, node- and service type-
user interface components required to handle some task, listing are becoming less useful as they provide a mere view
based on the state of the system. to the technical structure under MaCI’s abstraction.
The development is supported by a framework, which is Gimbo itself is fully independent of the applications of the
also constantly developed. The framework provides tools network, when started Gimbo connects to GIMnet after
for network browsing and launching the user interface which it automatically searches and constantly updates
components. Currently, there are C++ libraries for this, as the available services. The actual use of applications,
well as an user interface application (called Gimbo) which such as operating robots, is achieved with user interface
provides this functionality. The launching of components components that Gimbo can instantiate and configure.
is based on predefined command line parameters as well Available components and their descriptions are shown as
as GIMnet messaging protocol. This makes the compo- a list. Already running components are also visualized and
nents independent of the rest of the framework (One can a user may assign new services into them at will. Services

135
TA 2010
Timisoara, Romania, Oct 5-8, 2010
Fig. 2. Component instantiation process: Gimbo passes
connection parameters to the component on command
line, after which all communication is performed on
GIMnet.
are assigned to components by selecting them from service
listing and selecting a target component. Services may be
assigned individually, or using a branch of a service tree.
4.4 Component architecture
Components are individual programs that implement user
interface functionalities for the system. Components may
be specialized for visualization of single service type, (e.g.
component for viewing image stream), or fuse multiple
types into more complex user interfaces. Components are
not only for visualizing data, but may also produce output
and be used to control other modules.
In order to a component program to be compatible with
Gimbo, it has to accept certain command line parameters
and provide a GIMnet interface for inter-component com-
munication. Command line parameters are needed for the
instantation of a component and they define the GIMnet
connection, GIMnet-id of the host program and an unique
id for each component. When a component has started,
it connects to GIMnet, using the given parameters. When
the connection is established, the component reports back
to Gimbo by sending its id. (Fig. 2) This way Gimbo
can bind a certain component process and its instance on
the network. The network identifier of the component is
carried in the message’s header. After Gimbo has received
the report message, it starts sending service assignments to
the component. A service assignment message contains de-
scriptions of the selected services. If a component receives
an assignment for a type of service it does not support,
the component just discards it.
Component instantiation is not limited to Gimbo. Every
program implementing the component protocol may load
components, thus allowing component nesting. To ease
development, Gimbo protocol’s loader and component
implementations are available in C++ classes which can
be easily added to other programs.
The behavior of the components is described in XML
files. A file contains component’s name, the location of
the executable file (on the user’s computer) and a list of
supported service types.
136
4.5 Creating applications with component framework
A user interface application, in this context, means a
specific set of components that needs to be assigned, con-
figured, and running in order to perform a certain task
with a certain system. In the presented framework an
application is created by connecting Gimbo to a GIMnet,
selecting the services required, executing the components
required by the services, after which the components are
configured to match the human-input-devices of the user
(e.g. configuring the type of joystic to specific joints of a
robot) as well as the displays of the user (by placing the
components into the screen as desired). This process, even
not complicated for an expert, requires tedious configura-
tion and deep knowledge of the system. In general this is
not desired feature from the usability point of view.
For this reason, Gimbo allows users to save their sessions,
i.e., connection parameters, currently running components
and services assigned to those components. Sessions allow
the creation of various applications using the variants
of generic components with different configurations. The
creation of session needs to be once (by an expert) after
which any user can use them. Currently, sessions provide
the highest level of abstraction of the system as user does
not have to know specific targets, but may directly start
the application.
5. CASE STUDY: CONTROL OF MULTI-ROBOT
SYSTEM
So far, Gimbo and various components have been mainly
used in controlling small work machines (Fig. 3) and
laboratory robots. The two work machines used, are diesel
powered, weighing around 1 ton, digitally controlled and
equipped with laser scanners, cameras and manipulators
for digging and moving sand. Performed tasks include
direct teleoperation, waypoint driving and digging. All
tasks have been performed outdoors or in a testing hall.
Smaller laboratory robots have similar setting of sensors,
but have been operated in indoor environment.
Despite the considerable contrast between used robot
platforms, the user interface has proven to be flexible
enough to implement required controls and still be usable
for both.
One of Gimbo’s component, MuRO (Multi Robot Opera-
tor) is a user interface for supervisory control and direct
teleoperation of multiple robots. It provides an overview of
the state of multiple robots, fusing their sensor information
to one display. Robots are drawn on their locations as icons
on a map. Sensor data is drawn relative to the robots or
displayed as small icons or gauges next to them. Robots
may be driven manually or assigned drive paths. The user
may assign MuRO a whole group of robots at once. From
the given set of services, MuRO automatically detects
individual robots and their devices.
When MuRO is used for teleoperation, other user interface
components are usually used with it. Typical additions are
image viewer component for displaying camera images,
joystick control for operating manipulators and turning
camera and an alarm display for a more detailed view on
messages from the robot. An example of a teleoperation
setting is given in Fig. 4.
TA 2010
Timisoara, Romania, Oct 5-8, 2010
Fig. 3. The two work machines used in our research.
Fig. 4. A typical setting of components for robot teleoper-
ation: components for camera view, joystick control,
alarm viewing and MuRO for viewing robots and their
sensor data on a map.
6. CONCLUSIONS AND FUTURE WORK
The framework presented in this paper is designed to
control a generic fleet of machines in specific applications
by using a generic components. Gimbo provides the user a
good overview on the system state and enables launching
of components for the services. It has turned out to be a
valuable tool for using and developing the system. Gimbo
has freed users from the cumbersome process of using
command line tools that require manual configuration of
multiple connection and service parameters. Using basic
137
functionalities of the system, such as driving a robot or
simply viewing its state or sensors, have become routine
procedures requiring only few clicks. This has clearly sped
up testing and debugging phases of developing process
where the need for visual feedback and frequent test runs
are common. The abstractions and mechanisms provided
by MaCI are essential prerequisite for this kind of user
interface.
A clear and intuitive user interface is also valuable when
demonstrating the system. Gimbo has helped new users
and developers to quickly familiarize with the system.
New developers can easily start implementing their own
application when they are provided with graphical tools
that help in debugging and allow them to see results imme-
diately. Currently, the framework includes more than ten
user interface components, including logging, playback,
image stream visualizators and more complex interfaces
such as Muro. The framework is continuously developing
and not all the design criteria have been me yet. Neverthe-
less, the framework at its current form enables to creation
of user interface applications for a generic robotic system.
6.1 Lessons learned and future work
The concept of generic user interface for a generic (robotic)
system poses some fundamental challenges that have been
met during the development process. An itemized list is
given here for further discussion and as lessons that have
been learned:
• Hardware abstraction is preferable for interface de-
sign, but a problem for user interface. One example
is the physical appearance of the robot, a generic
position interface does not reveal the size and looks
of the robot, yet this is relevant information for an
operator.
• At our current system, there can be several instance of
control interfaces (for example several levels of path
planners). Presently, the user needs to know which
one to use.
• Distributing the user interface among separate com-
ponents, causes problems in terms of usability. A
good example is the location of focus in computer’s
desktop: the window that is selected, receives user
inputs, such as keystrokes. The problem is raised
when the user thinks that the focus is somewhere else
than where it should be.
A solution to the problems caused by heterogeneity and
multiple levels of control, is to find a way for the robots
to describe their own user interface, or “dashboard”.
The description should include the interfaces that are
intended for the user as well as the description of the
robot’s properties. The next step towards this is to extend
Gimbo’s session format to describe component layout on
the screen and settings for specific components. Robots
could then provide these sessions for Gimbo over a specific
MaCI interface.
The MaCI abstraction is also going to need a way to define
a hierarchy among control interfaces. This would allow the
separation of devices to control level and user interface
level. Control level devices are the ones working uder
robots automatic control while user interface devices are
TA 2010
Timisoara, Romania, Oct 5-8, 2010

accessed through the UI. Problems caused by component

distribution are going to be approached by extending the
component protocol to support communication and event
sharing between components.
An interesting future experiment is to incorporate Gimbo’s
component instantiation functionality to components. As
an example, MuRO displays the state of a multiple ma-
chines on a map and enables the configuration of paths
for autonomous driving. In some cases, the robots face
situations that require human intervention. A convenient
way to execute human intervention (teleoperation) would
be to let MuRO start a separate component more suitable
for the problem.

REFERENCES
Ando, N., Suehiro, T., Kitagaki, K., Kotoku, T., and
Yoon, W.K. (2005). Rt-middleware: distributed com-
ponent middleware for rt (robot technology). In In-
telligent Robots and Systems, 2005. (IROS 2005). 2005
IEEE/RSJ International Conference on, 3933–3938.
doi:10.1109/IROS.2005.1545521.
Brooks, A., Kaupp, T., Makarenko, A., Williams, S.,
and Orebaeck, A. (2007). Orca: A component model
and repository. Software Engineering for Experimental
Robotics, 231.
Calisi, D., Censi, A., Iocchi, L., and Nardi, D. (2008).
OpenRDK: a modular framework for robotic software
development. In Proc. of Int. Conf. on Intelligent Robots
and Systems (IROS), 1872–1877.
Chaimowicz, L., Cowley, A., Sabella, V., and Taylor, C.
(2003). ROCI: A distributed framework for multi-
robot perception and control. In Proceedings of the
2003 IEEE/RJS International Conference on Intelligent
Robots and Systems, 266–271. Citeseer.
Gerkey, B., Vaughan, R., and Howard, A. (2003). The
player/stage project: Tools for multi-robot and dis-
tributed sensor systems. In Proceedings of the 11th
international conference on advanced robotics, 317–323.
Citeseer.
Kramer, J. and Scheutz, M. (2007). Development en-
vironments for autonomous mobile robots: A survey.
Autonomous Robots, 22(2), 101–132.
Lipski, C., Berger, K., and Magnor, M. (2009). vIsage-A
visualization and debugging framework for distributed
system applications. In WSCG 2009 Full papers pro-
ceedings.
Mohamed, N., Al-Jaroodi, J., and Jawhar, I. (2008).
Middleware for robotics: A survey. In 2008 IEEE
Conference on Robotics, Automation and Mechatronics,
736–742.
Mohamed, N., Al-Jaroodi, J., and Jawhar, I. (2009). A
Review of Middleware for Networked Robots. IJCSNS,
9(5), 139.
Saarinen, J., Maula, A., Nissinen, R., Kukkonen, H.,
Suomela, J., and Halme, A. (2007). Gimnet - infras-
tructure for distributed control of generic intelligent
machines. In Proceedings of the 13th IASTED Interna-
tional Conference on Robotics and Applications Telem-
atics 2007. The 13th IASTED International Conference
on Robotics and Applications Telematics.

138