J Sign Process Syst (2010) 59:33–43
DOI 10.1007/s11265-008-0272-9
SIGMA System: A Multi-OS Environment
for Embedded Systems
Wataru Kanda · Yu Murata · Tatsuo Nakajima
Received: 10 March 2008 / Revised: 5 August 2008 / Accepted: 2 September 2008 / Published online: 26 September 2008
© 2008 Springer Science + Business Media, LLC. Manufactured in The United States
Abstract Embedded systems are becoming increas-
ingly sophisticated and there exists a wide variety of
requirements such as traditional realtime requirements,
multimedia support, etc. It is hard to satisfy all of the
requirements by a single OS. Would they be satisfied,
the system would become complex and this would cause
new problems. A multi OS environment is an efficient
approach to deal with these problems and to satisfy
complex requirements while keeping the system sim-
ple. We propose a multi OS environment named the
SIGMA system targeted especially at multiprocessor
architectures. On the SIGMA system, guest OSes are
corresponded one-to-one with cores. As a result, op-
posing to existing multi OS environment using virtu-
alization techniques, the system does not degrade the
performance of the guest OSes. In addition, the guest
OS running on the SIGMA system requires almost no
modification to its source code.
Keywords Operating systems · Multi-OS
environment · Symmetric multiprocessors ·
Embedded systems
1 Introduction
Recently embedded systems, especially mobile phones
and consumer electronics products are becoming in-
creasingly sophisticated. These systems require rich
W. Kanda (B) · Y. Murata · T. Nakajima
Waseda University, Tokyo, Japan
e-mail: kanda@dcl.info.waseda.ac.jp
GUIs and multimedia support in addition to the tra-
ditional realtime processing capability. In addition, the
number of applications that are connected to networks
and provide services is increasing. To satisfy these re-
quirement, using a general purpose OS, particularly
Linux, which has a wealth of network applications,
middleware, device drivers and software development
tools, as a platform for embedded systems is starting to
gather attention.
However, most of the general purpose OSes are
designed and optimized for the common case. Their
design is optimized for the common case and not for the
rare case to increasing the system throughput. For re-
altime processing requirements in embedded systems,
worst case execution time should be take in to consider-
ation even when it is a rare case. General purpose OSes
are not always designed to preemptable even during
the execution of lower priority tasks. As mentioned
above, there is a conflict between realtime processing
requirements and high functionality requirements, so it
is hard for a single OS to satisfy both requirements si-
multaneously, as such systems tend to become complex
and unstable.
In order to deal with these problems, a multi OS
environment which runs a realtime OS and a gen-
eral purpose OS on a single machine has been sug-
gested. For example wombat [1]. For building multi
OS environment on a single machine, virtualization
technologies are popular in server and desktop field,
for example Xen [2]. However existing virtualization
technologies have a tradeoff between guest OS mod-
ification costs and the performance of the system. To
keep the performance overhead small, the guest OS
needed to optimized for the vartualization layer and the
code modification cost of guest OS increases. On the
34
other hand, to keep the amount of guest OS code modi-
fications small, the virtualization layer must emulate all
hardware instructions and its emulation cost increases
the performance overhead. Emulation degrades the
realtime response which is often crucial for embedded
systems. Thus, existing virtualization technologies that
are designed for server and desktop environments are
not suitable for embedded systems.
To cope with these problems, a system that enables
multi OS environment with a low performance over-
head and a minimal amount of guest OS modifications
is needed. In this paper, we propose a system which
achieves a multi OS environment on a single machine
without any specific hardware except a symmetric mul-
tiprocessor (SMP) that is generally found in any per-
sonal computer. The name of the system is SIGMA
system. The SIGMA system assigns one OS to one
of the processors. This approach achieves multi OS
environment with a minimum engineering cost and only
a little performance degradation. We ran a benchmark
software on its OS to evaluate the performance of
SIGMA system.
The rest of this paper is organized as follows. The
next section describes three virtualization techniques
as a related work which achieve multi OS environment
on a single machine. Section 3 describes requirements
for multi OS environments for embedded systems. In
Section 4 overview of the SIGMA system and detail of
its implementation are given. Section 5 shows evalua-
tion of the performance and the guest OS modification
costs. Section 6 discusses the aptitude of the SIGMA
system and related works against the requirements
described in Section 3. Finally, Section 6 summarizes
our research.
2 Related Work
Several virtualization technologies exist. The key el-
ement of virtualizaton techniques is the virtual ma-
chine monitor (VMM) which duplicates a single real
hardware interface to multiple virtual interfaces. This
interface is called a virtual machine (VM) and equips
all instructions(both privileged and unprivileged) of
processor and hardware resources(memory, I/O de-
vices,etc...) [3]. Basically, a program running on a
VM should behave as it was running on true hard-
ware except for some differences caused by the avail-
ability of system resources and timing dependencies
[4]. Virtualization technologies are generally catego-
rized into three approaches: full-virtualization, para-
virtualization, pre-virtualization.
J Sign Process Syst (2010) 59:33–43
2.1 Full-virtualization
Full-virtualization is achieved by complete emulation of
all instructions and hardware. The most significant ad-
vantage of full-virtualization is that almost all OSes can
be run on its VM without requiring any modification to
them. Since this approach emulates the hardware com-
pletely, the reliability of the guest OS is not degraded
by the virtualization. On the other hand, emulation of
the processor and hardware decrease the performance
of the guest OS and make the VMM massive. The
examples of VMMs developed for full-virtualization
are VMware [5–7] and Virtual PC.
2.2 Para-virtualization
Para-virtualization is achieved by modifying the guest
OS for a specific VMM. The performance of para-
virtualization is superior to that of full-virtualization
as the OS is optimized for the VMM. However, such
modifications extinguish some advantages achieved by
full-virtualization. For instance, it is impossible to run
a para-virtualized on different VMMs because the OS
is modified for only a single VMM. Examples of para-
virtualization hypervisors and its guest OSes are Xen
and XenoLinux [2], the L4 microkernel and L4Linux
[8]. L4Linux is a para-virtualized Linux which is ported
to run on a L4 microkernel. L4Linux uses the L4 mi-
crokernel as a hypervisor and is modified to run on
it. L4Linux is executed concurrently with other servers
in the unprivileged mode. Wombat [1] is also a para-
virtualized Linux which is ported to run on L4/Iguana
system. L4/Iguana consists of NICTA::L4-embedded
[9] and Iguana [10]. NICTA::L4-embedded is a L4
microkernel modified for embedded systems to support
realtime processing. Iguana is a collection of basic soft-
ware that provides OS services such as memory pro-
tection mechanisms and device driver framework for
embedded systems. NICTA::L4-embedded, Wombat
and Iguana have been developed at National ICT
Australia (NICTA).
2.3 Pre-virtualization
Pre-virtualization is achieved by modifying the assem-
bler codes of the guest OS [11]. The modification
consists of a multi-phase process called afterburning.
Since the afterburning is processed automatically, the
engineering cost is relatively small. Pre-virtualization
has the both benefits of full-virtualization and para-
virtualization, the low engineering cost and the high
performance. A pre-virtualized OS can support multi-
ple hypervisors such as Xen and L4 [12]. At present, a
J Sign Process Syst (2010) 59:33–43
single compiled image of a pre-virtualized OS can be
run on multiple types of hypervisor. Users can select a
hypervisor for any purposes [13].
3 Requirement
In order to build a multi OS environment for embedded
systems, following requirements are stated.
• code modifications to the guest OS should be
minimal
• the virtualization layer should be light
• it should be possible to reboot the guest OSes in-
dependently from each other
General purpose OSes like the Linux kernel are mas-
sive and complicated software and are frequently up-
dated to incorporate new features and to fix bugs, code
modifications should be kept small or it will get hard to
go along with up-to-date kernel. From the viewpoint of
adding new guest OS to the environment ,the smaller
code modification also has an advantage. Thus, code
modifications to the guest OS should be minimal.
The second requirement is concerns the virtualiza-
tion layer. To minimize the performance degradation
of the guest OS, the virtualization layer must be very
light. Especially in embedded systems, the guest OS is
required to have realtime capabilities. Also to minimize
the virtualization layer effect to the worst case response
time of the guest OS. In addition, smaller system is
easier to test and verify.
The third requirement is for the system availability.
When a problem occurs in one of the guest OS and
its reboot is needed, the guest OS should be able to
reboot independently from the other guest OSes. By
rebooting the guest OS independently, other systems
can provide services continuously and availability of the
whole system can be improved.
As mentioned in Section 2, both full-virtualization
and para-virtualization have a trade-off between the
code modification of the guest OS and the performance
degradation of the guest OS. The full-virtualization
approach hardly requires any modifications to the guest
OS, while it degrades its performance. The virtual-
ization layer also becomes large. On the other hand,
the para-virtualization approach minimizes the perfor-
mance degradation of the guest OS, while imposing
quite a lot of code modifications for the guest OS.
Thus, it is difficult to simultaneously satisfy the first two
requirements by the existing virtualization approaches.
35
4 Design & Implementation
This section explains the design and implementation of
SIGMA system, that provides the multi OS environ-
ment and is designed for multiprocessor architectures.
The SIGMA system does look like a VMM as it is a
multi OS environment, but it does not provide virtu-
alized hardware interfaces to OSes running on it. An
OS in the SIGMA system runs directly on the physical
hardware. The principal building blocks of the SIGMA
system are an interprocessor interrupt (IPI) [14] and an
interOS communication (IOSC). The system is imple-
mented on IA-32 architecture.
4.1 Approach and Overview
We assume that for most embedded systems, the num-
ber of OSes which are required to have high functional-
ity is limited. In most cases, we expect that the number
of guest OSes is two: a realtime OS and a general pur-
pose OS. Cases using more than three different OSes
are very rare. On the other hand, multicore processors
are becoming major and two or four processor in a
single machine is not a rare case. In this situation,
there are more processor than these are guest OS is.
Therefore SIGMA system assigns each guest OS an
own core so that the OS can use all CPU resources
exclusively and the OS can run in privileged mode.
Devices are divided among the guest OSes. In order to
share a device between OSes, the device must be man-
aged by one OS and other OSes can send and receive
data via communication mechanisms between OSes.
This approach has many advantages. First, resource
management mechanism is not needed and therefor
hardly no virtualization layer is required. This fulfills
the second requirement from Section 3: “the virtual-
ization layer should be light”. OSes running on the
SIGMA system can manage their assigned devices by
themselves, they act as if they were running on a single
normal hardware. And they can run in privileged mode
and use most of the privileged instructions. Thus, OSes
are hardly required to be modified and cat have as high
performances as the native system. This also fulfills our
first requirement: “code modifications to the guest OS
should be minimal”.
In Intel’s multicore architecture, one of the proces-
sors is selected to be a the bootstrap processor (BSP)
and the others are application processors (APs) during
the system initialization by the system hardware [15].
After configuring the hardware, the BSP starts the APs.
For the SIGMA system, we define that an OS running
on the BSP is a host OS, and that the OS running on the
36
Figure 1 The structure of the
SIGMA system.
Wombat
Iguana
Server
Server
...
NICTA::L4-embedded
CPU0 (BSP)
AP is a guest OS. Especially, we call the guest Linux,
“SIGMA Linux”. It is slightly modified for the SIGMA
system. In the SIGMA system, NICTA::L4-embedded,
Iguana and Wombat are running on the BSP, and
SIGMA Linux is run on the AP. The architecture of
the system is illustrated in Fig. 1.
4.2 Boot Sequence of the SIGMA Linux
All modules of the SIGMA system including the
SIGMA Linux kernel, are loaded to the memory at
initial boot time. The OS manager, one of the modules
running on the host OS, takes charge of starting the
guest OSes. Before the SIGMA Linux boots, the OS
manager acquires kernel information from L4. L4 has
information about the modules such as the location in
the memory, the image size and other parameters. In
addition, to define the memory regions available for the
guest OS, the OS manager creates new e820 memory
maps. The details of the memory management will be
described in the next paragraph. The OS manager also
writes the information that the SIGMA Linux uses such
as kernel parameters to a specified memory address. Fi-
nally, OS manager sends a startup IPI to the AP and the
SIGMA Linux starts booting. For the SIGMA Linux, it
seems as it was booted by the general bootloader such
as GRUB on a standalone system. For the guest OS,
the OS manager is something like GRUB.
4.3 Memory Management
Although a normal Linux kernel acquires a memory
map through a bios call to identify free and reserved
regions in the main memory, a Linux kernel running on
the SIGMA system obtains the memory map from the
OS manager.
Before booting the SIGMA Linux, the OS manager
executes a program that invokes the e820 function
on the AP and acquires the e820 memory map. The
function of the e820 is to check the memory size and
J Sign Process Syst (2010) 59:33–43
Manager
OS
...
SIGMA Linux
CPU1 (AP)
H/W
memory maps. Then, the OS manager modifies the
memory map to create memory spaces for the SIGMA
Linux. The modification is based on the pre-defined
memory regions which are available for the SIGMA
Linux. After the modification, the OS manager over-
writes the modified map to the memory and boots
the SIGMA Linux. Although the original native Linux
includes instructions that call the e820 function, those
are removed from the SIGMA Linux to prevent the
modified memory map from being overwritten again.
Finally, the SIGMA Linux reads the modified memory
maps and starts booting within the restricted memory
areas. This approach however produces a security prob-
lem where the SIGMA Linux might ignore the memory
restriction and access illegal parts of the memory. This
problem is discussed in Section 6.
4.4 InterOS Communication
The IOSC is a mechanism that lets OSes running
of different cores to communicate with each other.
The IOSC is a similar mechanism as the interprocess
communication (IPC) of a general OS. The IOSC is
processed by following steps.
(a) The source OS writes a message to a shared
memory region.
(b) The source OS sends an IPI to the destination
cores.
(c) The destination OS receives the IPI and reads the
message.
In the SIGMA system, each pair of OSes that
communicate shares two different memory regions
and each OS writes its messages to different ar-
eas. These memory regions are placed outside of
both the L4 microkernel and Linux. Their addresses
are specified statically when the system is built.
However these regions are not protected by any
mechanisms. Thus, it is possible for all guest OSes
to access these memory regions illegally. Because
J Sign Process Syst (2010) 59:33–43
all the guest OSes in the SIGMA system are al-
lowed to execute all privileged instructions, they can
change the memory mapping to the MMU of their
assigned processor and invade the memory regions
where the shared memory for IOSC is located.
If the IOSC is called frequently, the system per-
formance could be affected. This resembles IPCs of a
microkernel-based systems.
4.5 Device Management
Device management is mainly done by the host OS.
Although both the host and the guest OSes can use
devices directly, should some OSes use the same device
at the same time, a collision occurs. The SIGMA system
avoids these collisions by restricting device accesses to
the host OS.
Device drivers are implemented as device servers of
the host OS. A device access of a process in the host
OS is performed by just calling the device server. When
a process of a guest OS wants to use a device, the
process has to send a request to the device server of
the host OS by calling the IOSC service. This process
is performed by a stub driver in the guest OS. They
are called pseudo device drivers. Since the interface
of the pseudo device drivers are same as those of the
common device drivers, the process can use devices
without knowing the location of the particular device
and its driver.
4.6 SIGMA Linux Modifications
The SIGMA Linux is able to execute all of the hard-
ware instructions directly.These instructions need not
to be substituted with emulated codes or hypervisor
calls. For the SIGMA system, two kinds of the modi-
fications are needed. The first category consists of the
modifications related to memory regions. Following
three parts of Linux kernel were modified.
• The address that kernel setup routine is located was
changed
• The address that uncompressed kernel image is
located was changed
• The instruction to call e820 function was omitted
The memory region were the non-modified kernel
setup routine and uncompressed kernel image are lo-
cated is already used by the L4 microkernel. To avoid
overwriting the memory, the first two modifications
are required. The third modification is required for
constrict memory regions used by the SIGMA Linux.
37
The second category is interrupt configurations. In
the SIGMA system, guest OSes are not allowed to
share devices and interrupts. Thus, all interrupts must
be delivered to the corresponding guest OS properly.
In the Intel multiprocessor architecture, it is possible
to distribute interrupts among processors by interrupt
numbers. This mechanism is achieved with IO APIC
[14] functions. We modified the part of the SIGMA
Linux code which initializes the IO APIC to prevent
it from configuring anything set by the host OS being
overwrote by SIGMA Linux. Hereby, interrupts for L4
microkernel is delivered to L4 microkernel and inter-
rupts for Linux is delivered to Linux. To achieve this
mechanism, following options have to be selected at
Linux kernel configuration and Linux has to be forced
to use IO APIC.
• Local APIC support on uniprocessors
• IO-APIC support on uniprocessors
5 Evaluation
This section describes the evaluation of our prototype
system. We ran a collection of benchmark programs
to evaluate the performance of the guest OS. We also
examined the collision on shared memory between the
host OS and the guest OS. All experiments described in
this paper were performed on a PC with a 2.4GHz Intel
Core2 Duo with 32KB L1 cache for each core, 4MB
shared L2 cache, 512MB of RAM, and an ATA disk
drive. We used Linux kernel version 2.6.12, and 2.6.16
for Xen.
5.1 LMbench
We used the LMbench [16] to measure their perfor-
mance. LMbench is the micro benchmark software to
measure the performance of Linux. LMbench measures
the performance of system calls, context switches, la-
tencies of communications and so on. Since LMbench
uses regular Linux system calls, it can be suitable to
compare many Linux systems.
SIGMA Linux is compared to three systems: vanilla
Linux kernel running on a single core (Native Linux),
vanilla Linux kernel running on two cores (Native SMP
Linux), and modified Linux kernel running on Xen
VMM (XenoLinux).
Figure 2 shows the cost of the interactions between a
process and the guest kernel. The benchmark examines
Linux system call and signal handling performance.
Any of the items of SIGMA Linux is the same as that
of native Linux, while XenoLinux takes longer time
38
Figure 2 LMbench:
Processes—time
in microseconds.
than native Linux in most of the cases. Some experi-
ments of SIGMA Linux made better results than native
Linux. This can be caused by the small difference of the
environment: SIGMA Linux ignores some interrupts,
but native Linux doesn’t.
We evaluated the cost of a process context switch in
the same set (Fig. 3). The cost of a context switch on
SIGMA Linux is also almost the same as that of the
native Linux and less than that of the native SMP Linux
and XenoLinux in any case.
The last benchmark evaluates the cost of the file
operations (Fig. 4). Xen is as fast as native Linux and
SIGMA Linux in file create/delete operations. On the
other hand, native SMP Linux is slower than the others.
The result shows that SIGMA Linux performs as well
as native Linux. It means that there are very few over-
heads in SIGMA Linux comparing to native Linux.
From the all results, SIGMA Linux presented as
the almost same performance as native Linux. In ad-
dition, compared to native SMP Linux and Xeno-
Linux, SIGMA Linux shows much better performance
than them.
If multiple threads are appropriately assigned to
processors and there are no dependencies between
the threads, native SMP Linux can achieve efficient
Figure 3 LMbench: Context switching—time in microseconds.
J Sign Process Syst (2010) 59:33–43
processing. Although it may be suitable for math calcu-
lation fields and image processing to introduce native
SMP Linux, it is not always efficient way for general
purpose use. To create a program which is optimized to
parallel processing, a special compiler may be required
such as automatic parallelizer and it may be also re-
quired to use a parallel programming language such as
Fortress [17]. As described above, native SMP Linux
often shows slower performance than native Linux on
a single processor. There are some possible reasons.
The first is lock contention: if one processor has already
acquired the lock of the critical sections, the other
processors have to wait until it will be unlocked to
access the regions. In this case, the advantage of parallel
execution cannot be made the best use of. The second
is the memory access collision issues, which will be
described in Section 5.2. The third is cache inconsis-
tency. In a multiprocessor system, when one processor
changes a page table or page directory entry, the change
must be propagated to all the other processors to keep
the cache consistency. This process is commonly re-
ferred to as TLB shootdown and performed by IPI [15].
TLB shootdown in IA-32 architecture must guarantee
either of the following two conditions: different TLB
mappings are not used on different processors during
Figure 4 LMbench: File & VM system latencies in microseconds.
J Sign Process Syst (2010) 59:33–43
updating the page table, the OS is prepared to deal with
the case where processors use the stale mapping during
the table update. A processor can be stalled or must
invalidate the modified TLB entries in order to keep
the TLB consistency. As a result, this may degrade the
system performance.
XenoLinux shows more than two times slower than
SIGMA Linux and native Linux at worst. The most
substantial element is the emulation of privileged in-
structions handled by Xen hypervisor. Many of IA-
32 instructions related to memory and interrupt and
I/O management must be performed in the privi-
leged mode. To perform those operations, XenoLinux
invokes hypervisor, which actually handles such in-
structions. Such invocations are called hypercalls. The
hypercall is very expensive processing. Although some
optimizations are implemented into Xen to reduce the
cost of the hypercall, for example, some hypercalls are
handled together, it can’t be enough to improve the
performance degradation substantially.
On the other hand, SIGMA Linux shows as fast
performance as native Linux. The most significant ad-
vantage of avoiding performance degradation is that
the guest OS can handle the privileged instructions
directly. The configurations of page tables and interrupt
vector tables, enabling/disabling interrupts, device con-
trols require privileged instructions and those are usu-
ally performed by emulations in general virtualization
approaches. Though, SIGMA Linux hardly requires
such emulations except in using device drivers run on
the other OSes. In addition, the VMM of general virtu-
alization manages the guest OSes and usually handles
scheduling of the guest OSes. In other words, the VMM
must decide one of the guest OS to run and save and
restore all OS contexts. This context switches between
OSes are similar to that of processes in general OS and
may degrade system performance. In SIGMA system,
multi OS run concurrently and scheduling of OSes is
not required at all. The superiority of SIGMA Linux
in performance aspects is based on the architecture
of SIGMA Linux which is almost same as that of
native Linux.
5.2 Memory Access Collision
Memory accessing can conflict at a memory bus in
shared memory multiprocessor architecture. The
conflict causes the performance degradation even in
SIGMA system. When more than two OSes access
to the memory concurrently, while one OS performs
memory related operations, the others can be waited
for it to become available. We call such conditions
memory access collisions and the performance may
39
become slow in such cases. Fortunately, many kinds of
multiprocessors have cache memory in each processor
and a processor can access cache memory much faster
than memory. If a process accesses data exists in cache
memory, the read/write operation becomes faster. On
the other hand, if the data isn’t in cache memory, the
process has to access main memory. Such operations
are expensive compared to the cache memory. Thus,
the results of memory latencies depend on whether
the target data is in cache or not. We measured the
latencies with memory access collision in five different
conditions described in Table 1. The main purpose of
the evaluation is to survey how much affection there
are from activities of other OSes which share same
memory bus.
The five conditions are categorized by cache avail-
ability and memory stress. “No Cache” and “Both No
Cache” condition mean that caches in each processor
are disabled. In “Stress”, to generate loading condi-
tion, the program which only repeat read and write to
memory is run on the host OS, Wombat. In that case,
the memory access of the guest OS in “No Cache”
condition can be sometimes affected. In “No Cache
with Stress” condition, the delay is more increased
because the many access from the host OS occurs. In
“Both No Cache” and “Both No Cache with Stress”
condition, the memory collision is assumed to be much
more increased because all processors try to use the
memory bus in reading or writing to the memory. The
evaluations are performed on the guest OS and we
used LMbench as an evaluation tool. The results are
illustrated in Fig. 5. Figure 5 shows that, in both cache
enabled and disabled configurations, the performance
of the guest OS in loading condition degrades ten
to fifteen percents compared to that in unloading. In
addition, when loading size is below four megabytes,
the latencies of “No Cache” and even “No Cache
with Stress” conditions are same as the “Normal”.
That is, no performance degradation occurs below four
megabytes when the caches are enabled on the target
processor.
Table 1 The measurements condition of memory access collision
in SIGMA system.
Condition Host OS Guest OS
Cache Stress Cache
Normal Enabled No Enabled
No cache No No Enabled
No cache with stress No Stressed Enabled
Both no cache No No No
Both no cache with stress No Stressed No
40
Figure 5 LMbench:
lat_mem_rd: Memory read
latencies in the guest OS
with memory collisions.
In general IA-32 architecture, processors and mem-
ory are connected by the 32-bit system bus, and the bus
is shared by all processors on the system. Therefore, if
multiple processors try to access the memory, the mem-
ory access may conflict. In the architecture, memory
reading/writing is performed in accordance with mem-
ory ordering model of the processor, which is the order
in which the processor issues reads and writes through
the system bus to system memory. For a example, in
write ordering of multiprocessor system, although each
processor is guaranteed to perform writes in program
order, writes from all processors are not guaranteed
to occur in a particular order [15]. Therefore, if the
system memory becomes loaded condition, multiple
processors may access to the memory at the same time
and actually waiting time can be increased for each
processor. As a result, those conflicts caused such delay
shown in Fig. 5.
On the other hand, the reason why such delay didn’t
occur in loading four megabytes or less is related to L2
cache size of the processor. If the loading data is in the
cache, the processor need not to access the memory
and the accessing does not affected by the memory
ordering. Therefore, if the each processor has more
caches, the possibility of the cache hit increases and it
minimizes the delay by the memory access competition,
as a result.
Table 2 The engineering cost System Modified lines
of the guest OSes.
SIGMA Linux 25
XenoLinux 1441
L4Linux 6500
J Sign Process Syst (2010) 59:33–43
5.3 Engineering Cost
We compared the number of modified lines of each
guest OS to measure the development cost quantita-
tively. One of the principle purpose of SIGMA system
is to minimize the engineering cost of the guest OS as
small as possible. The results are shown in Table 2.
Table 2 indicates that the modification cost of
SIGMA Linux is much smaller than that of XenoLinux
and L4Linux. A significant reason is that SIGMA Linux
has the almost same structure as native Linux. That is,
SIGMA Linux can execute the instructions that control
the hardware directly, and such instructions need not
to be substituted with emulated codes and hypervisor
calls. The required modifications in SIGMA Linux are
the memory related issues described in Section 4.3 and
interrupt configurations and hardware checking parts.
6 Discussion
As mentioned in Section 1, a multi OS environment is
an effective approach for building systems that require
a rich GUI and multimedia support in addition to
traditional realtime processing capability requirements.
Therefore the need for multi OS environments for
embedded systems is increasing. In this section, we
classify multi OS environment architectures into
three models shown in Fig. 6. Then, we discuss their
fulfillment of the six requirements we have laid. The
first three requirements are modification, performance
and independent reboot described in Section 3. Next
two requirements, protection and realtime capability
are also important requirements for most embedded
J Sign Process Syst (2010) 59:33–43
Figure 6 Three types of
multi OS environment (a–c).
App
App
OS0 OS1
VM0 VM1
VMM
CPU0 CPU1
H/W
(a)
systems. At last, we discuss about the number of the
guest OSes that should be able to be run in parallel.
The brief results are shown in Table 3 and details are
described in the rest of this section.
6.1 Multi OS Environment Models
Existing multi OS environments that are achieved by a
virtualization techniques are basically divided into two
models. We added the model of the SIGMA system to
these two models and classify the models of multi OS
environments into three varieties shown in Fig. 6.
1. Model1 This model realizes a multi OS environ-
ment by using a VMM. As shown in Fig. 6a, mul-
tiple OSes runs on VM interfaces provided by the
VMM. In this model, all OSes run as guest OSes.
Each OS is scheduled and inter OS communica-
tion and device sharing mechanisms are provided
by the VMM. It also provides memory protection
mechanism between each OS and the VMM. Major
examples are Xen and VMware server.
2. Model2 This model provides multi OS environ-
ment by executing the guest OS as a process in
the host OS. The model is illustrated in Fig. 6b.
There are two approaches in achieving a multi OS
environment with this model. One is running a
para-virtualized guest OS as a process in the host
OS. L4Linux and Wombat is a typical example
of this approach.There also RT-Linux [18] and
Linux on ITRON [19] that focus on the realtime
execution. Another approach is building a com-
pletely virtualized hardware environment on the
host OS by using emulators such as QEMU [20].
Table 3 Comparison of Model Modification Performance
three types of multi OS
environment. TYPE1  
TYPE2  
TYPE3  
41
App
App
App
App
OS1
OS0 OS0 OS1
CPU0 CPU1 CPU0 CPU1
H/W H/W
(b) (c)
In both approaches, the guest OS is scheduled as
a process in the host OS. Inter OS communication
and device sharing mechanisms are implemented
with functions provided by the host OS. Memory
protection mechanisms between processes in the
host OS are provided by the memory protection
mechanism between the guest OS and the host OS.
3. Model3(SIGMA system) This is the approach that
we have proposed in this paper and illustrated in
Fig. 6. The SIGMA system assigns an individual
CPU to each OS and the guest OSes. Therefore all
the guest OSes can use all CPU resources exclu-
sively and no virtualization layer exists under the
guest OS. Thus, processes of each OS are scheduled
completely independent from other OSes. The de-
vice sharing mechanisms are provided by a server
in the host OS. However, in order to minimize
overhead and code modification costs of the guest
OS, the host OS and the guest OS are running in
privileged mode. Thus it is hard to provide any
memory protection mechanism between the guest
OSes and the host OS.
6.2 Modification and Performance
The first two requirements are the low modification
cost and the performance of the guest OS. In Model1
and Model2, virtualization techniques are used to
run a guest OS. However full-virtualization and para-
virtualization have a trade off between the modification
cost of the guest OS and the performance of the guest
OS. Thus, a compromise between the two has to be
made. On the contrary, Model3 hardly requires any
modifications to the code of the guest OS.
Reboot Protection Realtime Guest OS Number
   
   
 ×  ×
42
6.3 Independent Reboot
The third requirement is the capability to reboot the
guest OSes independently. On Model1, guest OSes
are booted and rebooted by functions provided by the
VMM. Thus, all OSes can be rebooted independently.
On Model2, the guest OSes can be rebooted like any
process in the host OS. However, since guest OSes
strongly depends on functions of the host OS, it is
impossible to reboot the host OS independently. In
Model3 there is no strong dependency between the
guest OS and the host OS. However boot and reboot
of the guest OSes are done by functions provided by
the host OS. Thus, reboot of each OS can be done
only by the host OS. That is, all OSes can be rebooted
independently only if the host OS can execute functions
to reboot each OS.
6.4 Protection
Next requirement is a memory protection mechanism
between the guest OSes and the host OS. Generally in
the Model1 approach, VMM is executed in privileged
mode and guest OSes are executed in nonprivileged
mode with different address spaces. Therefore,
memory protection can be provided between the guest
OSes and between the guest OS and the host OS.
In Model2, the memory of the guest OS is protected
by the memory protection mechanism that the host
OS provides to its processes. If the host OS does
not provide any memory protection mechanism to
its processes, it will be difficult to provide memory
protection between the guest OSes. If any memory
access violation occurs in the kernel of the host OS,
it may be impossible to protect the guest OSes from
that error. It is also impossible to provide any memory
protection mechanism in the SIGMA system. Because
all OSes in the SIGMA system run in privileged mode
and are allowed to execute all privileged instructions,
the guest OSes of the SIGMA system can change any
memory mappings and invade the memory regions that
should be managed by other OSes.
6.5 Realtime Capability
Next requirement is the real-time capabilities generally
required from embedded systems. Existing VMMs are
designed for server and desktop environments and they
are not meant to be used in embedded systems. Para-
virtualization techniques try to minimize virtualization
overhead. However, it is difficult to guarantee certain
real time capabilities of a guest real time OS with
Model1. Similarly with Model2, it is hard to guarantee
J Sign Process Syst (2010) 59:33–43
certain realtime capabilities of a guest OS because of
the virtualization. However, it is not difficult to guaran-
tee realtime capabilities of the host OS, since the host
OS is running on physical hardware and there is thus no
performance degradation. Therefore, if the host OS is
a realtime OS, the system is able to run realtime tasks.
All OSes in Model3 run on physical hardware and have
almost the same performance as a native OS would
have. That is, realtime OSes can be executed with no
performance degradation in Model3.
6.6 Number of Guest OSes
Last requirement is on the number of guest OSes that
are possible to be run on the system simultaneously.
Model1 and Model2 have no limitation to it. Since
SIGMA system assigns a whole CPU to one guest OS,
the number of guest OSes in the SIGMA system is
restricted by the number of CPUs. Model3 cannot run
more guest OSes than there are CPUs.
7 Conclusion
We have proposed an asymmetric multi OS environ-
ment named the SIGMA system on SMP. Asymmetric
means that there are two types of OSes running on
separate cores: L4 microkernel and its servers, includ-
ing the OS manager, are used as a host OS, and a
modified Linux named the SIGMA Linux is used as a
guest OS. The SIGMA system can avoid performances
degradation of the SIGMA Linux because the SIGMA
Linux has direct control of its hardware.
The evaluation clearly showed that the SIGMA
system performs much faster than virtualization tech-
niques such as Xen and that it achieves the perfor-
mance of native Linux. The SIGMA system can be
realized with minimal overhead because no emulations
are required to handle privileged processing. We also
measured the latency of the SIGMA Linux memory
loading. The results showed that the SIGMA Linux
was little affected in the condition. In addition, the
engineering cost in implementing a SIGMA system
is much smaller than that of other para-virtualization
approaches.
Consequently, the experiments proved that SIGMA
system enables a multi OS environment with only a low
performance overhead and only small modifications.
We believe that the SIGMA system is adoptable to em-
bedded systems that require a rich GUI and multime-
dia support along with traditional realtime processing
requirements.
J Sign Process Syst (2010) 59:33–43 43

References

1. Barham, P., Dragovic, B., Fraser, K., Hand, S., Harris, T.,
Ho, A., et al. (2005). Wombat: A portable user-mode linux
for embedded systems. In Proceedings of the 6th Linux
Conf.Au Camberra. April 2005.
2. Barham, P., Dragovic, B., Fraser, K., Hand, S., Harris, T.,
Ho, A., et al. (2003). Xen and the art of virtualization. In
SOSP ’03: Proceedings of the 19th ACM symposium on op-
erating systems principles (pp. 164–177). Oct. 2004.
3. Goldberg, R. (1974). Survey of virtual machine research.
IEEE Coumputer, 7(6), 34–45.
4. Popek, G. J., & Goldberg, R. P. (1974). Formal requirements
for virtualizable third generation architectures. Communica-
tions of the ACM, 17(7), 412–421.
Wataru Kanda received B.S. degree of Engineering in Waseda
5. Rosenblum, M., & Garfinkel, T. (2005). Virtual machine
University, Tokyo, Japan in 2007 and currently working towards
monitors: Current technology and future trends. Computer,
a M.S. degree at Waseda University. His research interests are
38(5), 39–47.
embedded systems and virtualization.
6. Sugerman, J., Venkitachalam, G., & Lim, B. (2001). Virtu-
alizing I/O devices on VMware workstation’s hosted virtual
machine monitor. In: USENIX annual technical conference
(pp. 1–14).
7. Waldspurger, C. (2002). Memory resource management in
VMware ESX server. ACM SIGOPS Operating Systems
Review, 26(si), 181.
8. Härtig, H., Hohmuth, M., Liedtke, J., Schönberg, S., &
Wolter, J. (1997). The performance of microkernel-based sys-
tems. In SOSP f97: Proceedings of the 16th ACM sympo-
sium on operating systems principles (pp. 66–77). New York,
NY, USA.
9. Kuz, I. (2005). L4 User Manual NICTA L4-embedded API.
October.
10. Heiser, G. (2005). Iguana user manual (p. 4).
11. LeVasseur, J., Uhlig, V., Chapman, M., Chubb, P., Leslie, B.,
& Heiser, G. (2005). Pre-virtualization: Slashing the cost of
virtualization. Technical Report 2005-30, Fakult Nat f Nur
Informatik, Universit Nat Karlsruhe (TH). Nov. Yu Murata received M.S. degree of Engineering in Waseda
12. University of Karlsruhe Germany and University of New University, Tokyo Japan in 2007. His research interests are vir-
South Wales and National ICT Australia. (2005). Afterburn- tualization and embedded systems.
ing and the accomplishment of virtualization. April 2005.
13. LeVasseur, J., Uhlig, V., Leslie, B., Chapman, M., &
Heiser, G. (2005). Pre-virtualization: uniting two worlds. 23–
26 Oct. 2005.
14. Intel Corporation (1997). MultiProcessor specification ver-
sion 1.4.
15. Intel Corporation (2006). IA-32 Intel architecture software
developer’s manual.
16. McVoy, L. W., & Staelin, C. (1996). lmbench: Portable
tools for performance analysis. In USENIX annual technical
conference. Barkely (pp. 279–294), Jan. 1996.
17. Allen, E., Chase, D., Hallett, J., Luchangco, V., Maessen,
J.-W., Ryu, S. et al. (2007). The Fortress language specification
version 1.0 beta. Santa Clara: Sun Microsystems, Inc.
18. Yodaiken, V. (1999). The RTLinux manifesto. In Proc. of the
5th linux expo. Raleigh: NC, March 1999.
19. Takada, H., Iiyama, S., Kindaichi, T., & Hachiya, S. (2002).
Linux on ITRON: A hybrid operating system architecture for Tatsuo Nakajima is a professor of Department of Computer Sci-
embedded systems. In Proceedings of the 2002 symposium on ence and Engineering in Waseda University. His research inter-
applications and the internet (SAINT) workshops (p. 4). ests are embedded systems, distributed systems, and ubiquitous
20. Bellard, F. (2005). QEMU, a fast and portable dynamic computing. His research group is developing advanced software
translator. In Proceedings of the USENIX annual technical infrastructure for future many core processors for embedded
conference (pp. 41–46). April 2005. systems and future ubiquitous computing services.