You are on page 1of 11













TenAsys Real-time Hypervisor
Host Real-time and General-purpose Operating Systems on a Single Hardware Platform with Intel® Virtualization Technology
August, 2006

TenAsys Corporation 1400 NW Compton Drive, #301 Beaverton, OR 97006 USA +1 503 748-4720 fax +1 503 748-4730

Copyright © 2006, TenAsys Corporation. TENASYS, INTIME, and IRMX are registered trademarks of TenAsys Corporation. Other trademarks and brand names are the property of their respective owners.


the OS executes its instructions in supervisor-mode and its applications execute in user-mode. for the purpose of hosting an entire operating system (the guest OS) and its applications. The guest OS and applications running inside each virtual machine are native to the underlying processor instruction set. depending on the architecture of the OS and the nature of the driver. provides a means by which a VMM can efficiently host a 16-bit guest operating system (e. emulators have been written to run the code copied from ROM cartridges of old gaming systems on a desktop computer. referred to. alongside applications designed for the new OS and hardware. Also.” In this case a primary goal of virtualization was to allow legacy applications to run on newer machines. or is a subset of. 2006 . The most widely noted early examples are those implemented by IBM on their mainframe hardware giving their customers a means to easily upgrade from old “iron” to new “iron. part of the Intel® architecture since the 386 processor. running the iRMX® RTOS and a general-purpose OS side-by-side. in DOS RMX and the original iRMX® for Windows. ubiquitous on desktops and widely used for many embedded applications. Drivers might execute either in supervisor or user-mode. and the operating system calls from a real system. DOS and Windows versions thru ME). Most modern VMMs operate without the need for CPU instruction set emulation. some key hardware elements. Like an emulator.tenasys. instead of perfectly emulating a specific hardware platform. For example. page 2 of 11 Copyright © 2006.” Operating systems that utilize these execution modes are referred to as “protected-mode” operating systems. Ring 0 executes at the highest privilege level and ring 3 at the lowest. respectively. supports four distinct privilege levels of application execution named rings 0 thru 3. the typical VMM presents a virtual computing system that can be replicated multiple times on a single hardware platform and contains sufficient virtual I/O to support common client and server applications. In typical practice only two levels are ever used: rings 0 and 3. that is. Instances of these virtual real-time embedded applications are still in use today. the I/O devices inside a virtual machine need not emulate real hardware. A key difference between an emulator and a virtual machine is that a virtual machine can be built from virtual hardware. This x86 virtualization feature was exploited over fifteen years ago by TenAsys engineers. the underlying hardware. usually for the purpose of running applications developed for obsolete hardware on newer hardware.Virtualization Background Virtualization of computer hardware has been used for many decades. A machine emulator typically simulates the CPU instruction set. [1] Software that manages computer hardware virtualization is referred to as a Virtual Machine Manager (VMM) or • August. the typical emulator simulates an application’s operating system environment by intercepting legacy OS calls made by the emulated application and VM86 mode. on a single hardware platform. a VMM creates the illusion of a hardware platform. On an IA processor.g. as “supervisormode” and “user-mode. since the instruction set of the virtual machine is identical to. Emulation on the IA platform Intel® architecture (IA) processors. TenAsys Corporation • www. The VM86 hardware traps accesses to key processor resources so the VMM can maintain control of the machine hardware. [2] A VMM is akin to a machine emulator..

e. Static virtualization and real-time In 1997 TenAsys® Corporation introduced INtime®. single-platform” arrangement gives developers a means to build deterministic embedded Windows systems that can reliably control critical machine functions and simultaneously present high-level interfaces for system monitoring. All standard user interface I/O. as is Windows XP. an RTOS that runs deterministically alongside 32-bit and 64-bit versions of Microsoft® Windows® on a single IA hardware platform. such as the video.translating them into equivalent calls to the hosting OS. Running two specific protected-mode operating systems on a single platform is sometimes referred to as “static virtualization. Memory and I/O hardware designated for use by the INtime RTOS are “hidden” from Windows and dedicated for use by RTOS processes. A VMM. The emulated application runs in user-mode.. enterprise connectivity. the desktop OS which “shares” a single hardware platform with INtime.” [3] In this case Windows boots first and then INtime inserts itself between the system hardware and Windows. Copyright © 2006. Windows is unaware that the INtime RTOS and its processes exist. allowing Windows to run unmodified as the lowest priority task (i. One of the most difficult jobs a VMM must do is intercept guest OS instructions that modify supervisor-mode registers and data structures and then simulate the impact of those instructions inside the virtual machine on which the guest OS is running. INtime and Windows share a single hardware platform. hosts a complete protected-mode OS (the guest OS) plus its applications. remain under the ownership of Windows. The INtime RTOS is a 32-bit protected-mode operating system. 2006 page 3 of 11 . on the other hand. keyboard. making it very easy to trap and substitute system-level calls using the x86 ring • August. This “dual-OS. A protected-mode guest OS expects to run in supervisor-mode with full access to the underlying CPU registers and data structures. TenAsys Corporation • www. Windows runs as the lowest priority task (the idle task) in the INtime real-time task list. even as it is relegated to the lowest-priority task in the INtime task list. A unique form of virtualization makes this feat possible. the idle task) in the INtime real-time task list.tenasys. and mouse. and complex user interaction.

directly on the CNC machine. Austria. Australia puts an INtime plus Windows powered PC at the heart of their CNC machines. absent some form of cooperation between the VMM and the guest operating systems (sometimes referred to as “paravirtualization”) the VMM must resort to trickery.Static virtualization of an RTOS with Windows has many applications: • Phoenix Contact of Ann Arbor. Supervisor-level control can be reliably maintained by only one software system. a software-only VMM designed to support multiple protectedmode operating systems on an x86 virtual machine encountered significant challenges. The conventional VMM is targeted at solving problems for a corporate IT network: maximizing the use of server resources and simplifying the deployment and maintenance of client desktops. Like a general-purpose OS. and power trains. called Transaction Express. A few other generic I/O devices may be available to “install” inside the IT virtual machine. The “IT virtual machine” presented by a conventional VMM is not capable of supporting the deterministic scheduling and dedicated I/O required of real-time applications and their operating systems. most virtual machine I/O is limited to these basic devices. keyboard. on Windows. Despite these limitations. • AVL in Graz. The I/O presented by a conventional IT virtual machine usually consists of a CPU. 2006 . that is. RAM. mouse. Conventional IT virtual machine shortcomings The static virtualization method used by INtime succeeds where the conventional approach to virtualization used by traditional VMMs simply will not work. video. transmissions. however. but due to the difficulty associated with multiplexing a wide range of I/O between multiple virtual machines. • CNC maker ANCA from Melbourne. resulting in one of the fastest and most flexible Windows-based PLC engines for factory floor automation. for real-time applications. disk. this I/O complement satisfies a large percentage of IT server and desktop applications. AVL products are used by major car and truck manufacturers worldwide. drives their PUMA Open test bed software with INtime. resulting in a conflict between the VMM and the guest OS. Intel® Virtualization Technology Until the introduction of Intel® Virtualization Technology (Intel® VT) as part of the Intel® Core™ • August. and a network interface.tenasys. a conventional VMM must be “fair” in its approach to scheduling CPU time for virtual machines and allocating I/O resources among the virtual machines. TenAsys Corporation • www. The tricks a VMM must page 4 of 11 Copyright © 2006. Michigan implements their Steeplechase soft PLC engine on INtime and a PLC-to-enterprise network interface. making this form of virtualization popular with corporate IT groups as a means to quickly buildup and teardown server applications and client desktops. such as Windows or Linux. Both the VMM and the protected-mode guest OS expect to maintain supervisor-level control over the hardware platform. giving their customers fast and accurate machining along with the ability to model and preview grinding operations in 3D. PUMA Open is a real-time Windows test automation system for measuring and processing high-speed data from engines. The combination reduces their customer’s cost of operation by simplifying equipment requirements and avoiding expensive tool crashes during trial runs.

Also. requiring the VMM to emulate these instructions. The combination of multi-core CPUs and Intel VT are an ideal platform on which to move beyond multi-OS real-time systems that utilize static virtualization and build a real-time hypervisor based on dynamic virtualization. In those processors that include Intel VT an overarching operating-mode has been added. TenAsys Real-time Hypervisor Intel Virtualization Technology has enabled TenAsys to develop a hypervisor capable of supporting the demands of an RTOS while simultaneously hosting a general-purpose operating system (GPOS). problems exist in the area of address translation and interrupt masking where. involve modifying the guest OS binary code and running the guest operating systems at ring levels for which they were not written. both general-purpose and real-time. Likewise. by leveraging Intel VT. This second advantage means it is possible to restart one guest OS while other guest operating systems continues to run without interruption. where a hypervisor executes with final control of the CPU hardware. taking over complete control of the hardware platform. binary files of a guest OS may be modified to trap supervisorlevel CPU instructions. [4] Dynamic real-time virtualization Low interrupt latency. the dynamic virtualization implemented by a real-time hypervisor uses an “early start” technique.use. called VMX root. and the assurance that a VMM won’t “time slice away” the determinism and priority of real-time tasks are all key requirements of a real-time virtual machine. The downside to such VMM trickery is a decrease in performance and limited guest OS compatibility. 2006 page 5 of 11 . can be supported. one can build a hypervisor VMM that hosts protected-mode operating systems executing in ring 0 without giving up control of key CPU resources. A real-time hypervisor is a VMM that uses hardware virtualization technology to isolate and simultaneously host general-purpose operating systems and real-time operating systems. Intel VT provides a way for the VMM to implement virtual interrupts. like Windows or Linux. and the need to “fix up” binary files limits your guest OS options to those that have been certified for use with the VMM. A hypervisor that uses Intel VT can intercept key supervisor-mode operations executed by any software operating outside of VMX root without requiring a priori knowledge of the guest OS binaries or internals. For example. again. without Intel VT. TenAsys Corporation • www. Intel Virtualization Technology is designed to overcome the problems outlined above. Instruction emulation slows down the execution speed of the guest OS. This means that guest operating systems are allowed to “boot” only after the real-time hypervisor has constructed a virtual machine for them. an essential feature for hosting a real-time guest OS. direct access to specialized I/O. Further.tenasys. The boot sequence for each guest OS is under the control of the hypervisor. the software solutions result in performance overhead that cannot be tolerated by real-time applications. There are two significant advantages to building a multi-OS real-time system by using dynamic virtualization rather than static virtualization: A wide range of operating systems. the Copyright © • August. Unlike static virtualization (described above). Using this Intel VT hardware assist for virtualization.

In the simplest case. page 6 of 11 Copyright © 2006. protection. and interrupts. Using a real-time hypervisor to simultaneously support execution of a realtime OS and a general-purpose OS on a single hardware platform is quite different from installing a real-time thread scheduler. must be allocated by any VMM. by design.tenasys. inside a general-purpose OS kernel driver and subsystem solutions lack the context to apply Intel VT and insure isolation. TenAsys Corporation • www. The TenAsys real-time hypervisor leverages Intel Virtual Technology to partition processor resources between multiple guest operating systems. 2006 . and independence of operation between the RTOS and the GPOS. a conventional VMM evenly multiplexes these resources between the virtual machines.TenAsys real-time hypervisor enhances real-time application responsiveness and reliability in a “dual-OS. RAM. A key difference between the TenAsys real-time hypervisor and conventional VMM solutions is how physical resources are allocated to each virtual machine. single-platform” environment. A conventional VMM may allow priorities to be assigned to specific virtual machines. with even more precise control over interrupt latency and finer partitioning of I/O resources between multiple guest operating systems than previous solutions. such as CPU • August. in the form of a device driver or subsystem. When determinism and performance are more important than equal access. the processor virtualization features can be used to isolate resources for use by a specific virtual machine and its guest OS rather than to create virtual I/O for shared access between multiple virtual machines. Resources. I/O. but these “tuning parameters” only slightly modify the bias of a system that. attempts to fairly distribute physical resources among all the virtual machines on a given hardware platform.

an unnecessary and time consuming process. Even if only one virtual machine ever accesses a specific I/O device. This notion of applying I/O exclusively to a specific virtual machine is essential to guarantee real-time responsiveness. in order to more efficiently allocate limited physical RAM among multiple virtual machines. Without exclusive physical assignment of pertinent I/O you run the risk of waiting indeterminately for access to key devices. Graphics intensive applications need access to real hardware for maximum performance. must always be resident in physical RAM. However. the virtual machine containing the GPOS needs direct access to the physical frame buffer and its control I/O.. This approach requires a video helper function residing on the virtual machine that contains the physical frame buffer. mouse. Dedicating the physical video frame buffer to a single virtual machine guarantees optimum video features and performance for the GPOS and applications running inside that virtual machine. disk. fieldbus interface. depending on their video output needs. the wait can be significant. In a conventional VMM some or all of the memory in each virtual machine could be swapped to disk. showing the output of any one virtual buffer at a time on Copyright © 2006.Exclusivity for real-time To maintain the determinism of a guest RTOS in a virtual machine environment. with the output of the virtual video devices displayed in a window on the user interface of the virtual machine containing the dedicated video buffer. Similar arguments can be made for access to RAM. 2006 page 7 of 11 .g. and an enterprise Ethernet interface can be multiplexed and shared between all virtual machines. This is the only way to insure that every real-time event is serviced consistently. Specialized real-time I/O needs to be dedicated to its real-time virtual machine. The TenAsys real-time hypervisor guarantees that each real-time virtual machine is locked into physical RAM. should not be multiplexed between virtual machines. For example. with deterministic timing. and is never swapped to disk. For example. when a request is made to access that I/O a VMM that virtualizes I/O must translate the request by the virtual machine into real I/O accesses to the physical hardware. console. Ctrl-Alt-Fn) can swap between virtual displays. so the RTOS and application using that I/O can maintain real-time determinism and • August. and all of its real-time applications. Exclusivity for performance Exclusivity of I/O doesn’t apply only to a real-time virtual machine. a virtual frame buffer may be too slow and inadequate in features for an application that renders 3D moving images. such as a video capture card. For those systems where virtual frame buffers have sufficient performance and features for all virtual machines hosted by the hypervisor a simple keyboard switch (e. user interface I/O is usually not associated with time-critical events.tenasys. An RTOS. at the expense of leaving all other virtual machines with no output console. hardware that is specific to a real-time control application. or an Ethernet NIC designated for communication with real-time I/O devices. A serial console or a virtual frame buffer can be presented to the remaining virtual machines. the TenAsys real-time hypervisor distinguishes between resources that can be multiplexed by the VMM and those that are exclusive to a virtual machine. because it allows a real-time virtual machine to have direct physical access to its dedicated I/O. so devices like the keyboard. In that case. TenAsys Corporation • www. If another virtual machine has access to an I/O device. because the I/O is multiplexed.

such as a graphical user interface or access to an enterprise network. single platform” applications. new features can be added to the legacy application by simultaneously hosting a GPOS. these legacy applications may also be running on obsolete hardware or be in need of an update. The advantage. For example. they are reliable.g. facilitated by the real-time hypervisor. Three or more virtual machines can be hosted by the real-time hypervisor. The TenAsys real-time hypervisor can directly host legacy real-time applications or host a legacy RTOS and its application(s). to this fully virtualized video approach is that no video helper function is required. These legacy real-time applications continue to be used because they work. even if they share hardware devices on a single interrupt line. medical equipment. each containing a dedicated RTOS. and a DSP daughterboard dedicated to high-performance numeric algorithms. Interprocess communication. it is possible to simulate old hardware • August. a VMEbus system could be converted to a less expensive singleboard computer system by intercepting I/O requests to VMEbus I/O and redirecting it to equivalent on-board I/O devices. For example. Because I/O can be virtualized. an RTOS board implementing machine control. 2006 . a conventional system consisting of a general-purpose OS board serving the user-interface and enterprise nexus function. minimizing rewrite of proven legacy code. such as image processing. Legacy RTOS support Many systems exist today that depend on real-time code written many years ago. defense.the real monitor. insures that real-time interrupt latencies are minimized and that virtual machines cannot interfere with each other. The Intel VT hardware in an Intel Core microarchitecture CPU provides the TenAsys real-time hypervisor with the ability to “see” a hardware interrupt even if it has been masked by a guest OS. three hardware elements page 8 of 11 Copyright © 2006.. they are proven. This is an important feature. With minimal modification to existing code. Using real-time virtual machine technology. Multiple RTOS Support The use of the TenAsys real-time hypervisor is not limited to just “dual-OS. The ability to monitor interrupts at all times. for example. and aerospace). is used to augment the legacy application by adding new features via processes running in parallel virtual machines. Take. TenAsys Corporation • www. regardless of the state of their mask bits inside a virtual machine. and they may even be certified for an application (e. a single GPOS in one virtual machine and two real-time virtual machines. thereby enabling developers to maximize code preservation and maintain performance. Running legacy RTOS code in a virtual real-time machine is also how legacy real-time code can be migrated from obsolete hardware to modern embedded platforms. of course.tenasys. it means the real-time hypervisor can field hardware interrupts and insure that interrupts for a real-time virtual machine will always be serviced without delay. Unfortunately. Interrupt latency Control over hardware interrupts is substantially improved by Intel Virtualization Technology. Rewriting a proven and/or certified real-time application is rarely desirable or economical.

size. hosting three virtual machines. The key differences between multi-socket SMP platforms and single-socket multi-core platforms are cost. called the Intel® Smart Cache. Removing contention for resources in a multi-OS platform has a dramatic impact on realtime performance metrics. Multi-Core CPUs Multi-core processors are composed of multiple CPU cores in a single-socket package. one each for what was previously three separate (and expensive) pieces of computational hardware. Copyright © 2006. on a dual-core processor the TenAsys real-time hypervisor dedicates one CPU core to the RTOS and the other to the GPOS. With such low latencies real-time application control loops can execute in the 50200 microsecond range with very high precision.tenasys. Most general-purpose operating systems. The CPU cycles of the remaining core are used exclusively by the GPOS virtual machine. Intel Core microarchitecture processors include multiple CPU cores on a single silicon die. such as interrupt latency. [5] Intel multi-core processors combine the benefits of multiple execution cores on a single die of silicon. are designed to operate on and take advantage of SMP machines. such as Windows XP and Linux. symmetric multiprocessor platform (SMP). (Image courtesy of Intel) From a software application perspective. 2006 page 9 of 11 . reducing overall power consumption—an important consideration for embedded applications. Intel Core microarchitecture multi-core processors access their external bus via a shared leveltwo cache. Multi-core technology runs at lower clock speeds than single-core processors. maximizing performance and responsiveness of both operating systems. The multi-core architecture can deliver significantly greater performance at far lower heat dissipation than equivalent performance single-core processors. Multi-core processors mean faster scan times can be implemented and higher quality controllers can be deployed. TenAsys Corporation • www. multi-core systems look just like a multi-socket. TenAsys has measured a 10 to 1 improvement for interrupt latency on dual-core platforms compared to equivalent clock speed single-core platforms. Real-time virtual machine solutions derive maximum benefit from multi-core processors. Contention for key CPU resources. such as pipelines and the FPU. latencies measured in the 10-30 microseconds range on singlecore systems are reduced by an order of magnitude to 1-3 microseconds on dual-core systems.can be condensed into a single platform solution. eliminating inter-OS context switch times. providing real-time applications with 100% of the CPU instruction cycles of that core dedicated to the RTOS. a server hardware platform that has been available for many years. power consumption. These gains are of particular benefit to applications requiring high performance in a small embedded form factor. For example. and availability to the embedded market. Coordination between the RTOS virtual machine and the GPOS virtual machine is managed by the real-time hypervisor which uses built-in interprocessor interrupt mechanisms. all hosted on the TenAsys real-time • August. are avoided.

and video streaming and mixing applications. the PLC engine is assured of its need for continuous operation. regardless of the state of Windows. By dedicating a CPU core to each virtual machine. for example. provide disk logging. If necessary. Shared memory or a virtual Ethernet channel can be used to communicate between the two environments. performance. the basis of complex digital control algorithms. SSE instructions were introduced later and can efficiently handle floating point DSP operations and vector arithmetic. DSP Replacement Modern processors contain instructions designed to efficiently perform DSP arithmetic operations. for whatever reason. where each virtual machine has a dedicated CPU core to insure timeliness of response and ample performance. two hardware platforms would be employed because the Windows OS can guarantee neither continuous operation nor precision timing to satisfy the needs of the PLC engine. and space and simplifies development and communication between the two parts of the system: the Windows enterprise interface and the PLC control engine. Proven real-time applications can be hosted in a guest real-time virtual machine with little or no modification—enabling developers to maximize code preservation and maintain performance while using the real-time hypervisor to add new features via a parallel general purpose operating system. Windows is used as a man-machine interface to set important parameters. As a group these are known as SIMD instructions (Single Instruction. a Windows-based industrial control application where the real-time tasks run on a PLC engine that must operate continuously. Helper functions can be added to the real-time virtual machine to communicate with the GPOS virtual machine for coordinating between the existing legacy code and new feature code. Combining the two functions on a dual-core hardware platform saves time. Legacy Real-time Legacy real-time systems are a prime application for the TenAsys real-time hypervisor. the real-time integrity and performance of the PLC engine is also maintained. 2006 . without impeding operation of the PLC engine. such as Windows or Linux. TenAsys Corporation • • August.Application Scenarios Unlike the application of a conventional VMM to an IT server platform. This means that Windows can be rebooted.tenasys. obsolete hardware can be simulated because the VT-x technology allows the hypervisor to isolate I/O accesses. Because Windows and the PLC engine reside in separate virtual machines. The TenAsys real-time hypervisor hosts each part in a separate virtual machine. money. On Intel architecture processors the SIMD instructions are referred to as the MMX and SSE instructions. pattern recognition systems. Industrial Control Take. monitor system status. MMX instructions were introduced first and are limited to integer operations. These SIMD instructions are ideal for implementing digital filters. Multiple Data). page 10 of 11 Copyright © 2006. and act as an interface to the factory enterprise network. and security and would also benefit from the use of a GPOS. and with very precise timing. the TenAsys realtime hypervisor is best suited for control and data acquisition applications that require determinism. Normally.

pp. and real-time hardware subsystems onto a http://www. faster communication and coordination between RTOS and GPOS subsystems. vol. Real-time interrupt latencies are reduced by an order of [2] Answers. Computer Magazine. Conclusion The net gains from the application of TenAsys real-time virtual machine technology on Intel multi-core processor platforms are elimination of redundant computer and communication hardware. hardware platform. Complex real-time application loop cycle times in the 50-200 microsecond range are supported with very high precision and accuracy. improved reliability and robustness. real-time applications by dedicating a CPU core to the RTOS. and enterprise network [4] • August. and the FPU. August 2006. Intel Virtualization Technology in Embedded and Communications Infrastructure Applications.tenasys. 38.Rather than dedicate a costly DSP board along with an RTOS and/or a GPOS system to implement a complex control system it makes economic sense to combine these functions onto a single hardware platform. IEEE Computer Society. The result is an order of magnitude improvement in the quality and bandwidth of real-time control algorithms that can be deployed on a real-time Windows or Linux platform. and simplified development and debugging. Amit. are presented as virtual devices for use by all virtual machines. lowlatency.answers. Virtual Machine.163 [5] Intel Corporation. 2004.. http://www. http://doi. Intel® Advanced Platform Technologies. it is now possible to combine them on a single multi-core processor system. issue [3] Neumann. multi-core. pp. Using the TenAsys real-time hypervisor. 48–56. Contention for key resources. without the cost and complicated logistics associated with multiple separate pieces of Encyclopedia. Intel Virtualization An Introduction to Virtualization. et.. Dean. [1] Singh. I/O that can be shared. http://www. 2006 page 11 of 11 . an RTOS box for primary machine control. like keyboard. TenAsys Corporation • www. eliminating context switches between multiple operating systems. et. ca. Rich. The CPU instruction cycles of the RTOS core are available 100% of the time for its real-time applications. Coordination between multiple CPU cores is accomplished via built-in inter-processor communication mechanisms. re-use of proven legacy applications. The ability to reboot one OS while the other runs without interruption promotes significant cost savings by condensing systems comprised of separate GPOS. dedicating a CPU core to each function. such as CPU cycles. and a DSP board for high-performance data acquisition and filtering. are Intel Virtualization Technology is used to remove I/O contentions by isolating and dedicating those devices that belong to the RTOS for exclusive use by real-time applications.pdf Copyright © 2006.1109/MC. http://www. Where previously a GPOS system was used for human interface and enterprise network access. DSP. one could simultaneously operate three virtual machines. May 2005. 2-3. from 10-30 microseconds down to 1-3 microseconds. January 2006.kernelthread. mouse. Intel Technology Journal. Multi-core processors easily support multiple operating systems and high-performance.ieeecomputersociety.