Rootkits and the

OS Friendly Microprocessor Architecture

Patrick Jungwirth, PhD a
Thomas Barnett, Jr.b
Abdel-Hameed Badawy, PhD c
a
US Army Research Lab, Aberdeen Proving Ground, MD 21005
b
US Army Aviation and Missile, Research, Development and Engineering Center
Redstone Arsenal, AL 35898
c
Klipsch School of Electrical and Computer Engineering
New Mexico State University, Las Cruces, NM 88003
Army High Performance Computing Research Center

ABSTRACT
We examine how the hardware level security features in the OS Friendly Microprocessor Architecture improves
cybersecurity against a rootkit attack. A rootkit (root + kit) is a malicious program or tool -“kit” of programs designed to
obtain “root” level privileges (root for Unix, admin for Windows). Rootkits operate at the same security ring level as an
operating system. This gives rootkits access to kernel level data structures. Even with state-of-the-art security
technologies, it is very difficult to detect a rootkit. Rootkits have been used for digital rights management and copy
protection; however, the 2005 CD copy protection scandal illustrates how poor computer security can leave an open door
for other malware.
We present a security model of the OS Friendly Microprocessor Architecture and we present a short introduction to
rootkits. For this paper, we will focus on OS-kernel level rootkits. We will illustrate how the hardware security features
of the OS Friendly Microprocessor Architecture increases the difficulty for rootkit malware to compromise a computer
system.
Keywords: Microprocessor, OS, cyber security, rootkit, privacy

1. INTRODUCTION
We present a short introduction to cybersecurity covering the motivation for the OS Friendly Microprocessor Architecture.
“Cyber” security history examples provide illustrations of poor system security and some ideas for more secure systems.
Back in 1975, “Security Kernel Evaluation for Multics,” [1] presented a forward thinking definition of a secure system:
“A system is secure if it is Known to prevent all actions defined as unauthorized by the specification of its security
properties.” We point out system complexity is the bane of system security. We are interested in a simple, hardware
security mechanism that completely separates control from untrusted “data.” In this paper, we are going to look at the
hardware computer security mechanism in the OS Friendly Microprocessor Architecture to protect against rootkit attacks.
Cyber security can be compared to defending a medieval castle. The attacker only needs to find one weakness
(vulnerability) to get in. After a decade long siege on a seemingly impenetrable fortress, the Greek army devised a “social
engineering” attack and gained entry into Troy with the “Trojan Horse” [2]. A cyber-attack starts be gaining entry into a
system. The ultimate goal of a cyber-attack is to achieve administrator privileges without being detected. Administrator
privileges for a cyber-attacker is equivalent to Dark Lord Sauron’s ring: “O ne Ring to rule them all …” [3].
Current cyber security practices can be compared to in-band signaling. With in-band signaling, the system hacker already
has gained access. The 1970’s telephone network [4-5] used in-band signaling for control signals (tones). In-band
signaling combines voice (data) and control signals on the same telephone circuit. A blue box [6] provided the control
codes for the telephone network. Any hobbyist with electronics knowledge could easily build a blue box to control the
telephone network. In-band signaling could not tell the difference between the telephone service provider and a prankster.
Control information needs to come from a trusted source and/or be authenticated.

Cyber Sensing 2017, edited by Igor V. Ternovskiy, Peter Chin, Proc. of SPIE Vol. 10185,
1018503 · © 2017 SPIE · CCC code: 0277-786X/17/$18 · doi: 10.1117/12.2258235

Proc. of SPIE Vol. 10185 1018503-1

A good practical model for cyber security is the Post Office. A letter is addressed to a delivery address. The letter is
scanned to ensure there are no hazardous materials present in the letter. The Post Office is not interested in the message
inside the letter. The Post Office verifies the address is valid. The Post Office checks for sufficient postage. The letter is
transported and delivered. The actual path the letter takes from source to destination address is not important. The
important steps are the letter was scanned and the destination address was determined to be valid. We want the letter to
be routed based on the policies of the Post Office; not on instructions inside the letter. We do not trust “mobile code”
inside a letter. This simple model does not consider the insider threat.
Software system complexity, e.g. lines-of-code, is the nemesis of computer security. The fewer lines of the code, the
easier it is to achieve a high computer security assurance level. The highest assurance level, EAL7 [7], requires a full
proof-of-correctness. Security testing cannot achieve test coverage to this level of rigor: “… to prevent all actions defined
as unauthorized … ” Formal methods (proofs-of-correctness) are required to achieve this level of security. In practice,
large software programs cannot be fully and formally proven correct [8-11]. Secure micro-kernels with limited
functionality (compared to a modern, large OS) have been proven correct. The seL4 operating system is a recent example
of a formally proven microkernel operating system. The seL4 microkernel has 8,700 lines of C code and 600 lines of
assembly code [12]. High assurance separation kernels currently use: hypervisor techniques, hardware assisted
virtualization, and hardware assisted page tables to provide separation and isolation between OS and applications [13].
Researchers need to focus on simple hardware features that could drastically reduce the number of lines of software codes.
High security vendors and government agencies need to realize these complexities pose an intractable problem for controlling
costs of security evaluations and maintaining acceptable trustworthiness levels in their computing systems. As the cyber
warfront escalates, lowering the quality and assurance of high security systems poses a significant threat.
Separation Kernels Enable Rapid Development of Trustworthy Systems [13]

There is a rift between rival camps in the computer world. In the 1980’s, Gulliver’s Travels met the computer world.
There are two rival computer memory formats: big endian and little endian named by Cohen [14] after Gulliver’s Travels
and which end of the egg do you break first. In the 90’s up to today, there are two rival computer platforms: Windows
and Apple. The authors believe there is a similar disconnect between microprocessor architectures and Operating System
developers. Microprocessor architectures always lead the OS developers by several years. Computer security does not
have a high priority with the commodity microprocessor offerings. More research into simple hardware features that can
significantly reduce OS and software complexity is needed. We need a more balanced approach to hardware and software.
Both defense (computer security) and offense (system hackers) are moving computer security closer to the system
hardware level. In this paper, we will look at the hardware security mechanism in the OSFA. We will illustrate how the
hardware security mechanism defends against rootkit attacks.

2. ROOTKIT HISTORY
‘Root’ is the highest privilege level inside an operating system. Root has complete access to a computer system’s
hardware, memory and software. For an operating system to manage a computer’s system resources, it requires root level
access. ‘Kernel’ is the general term referring to the core of an operating system with root level access.
A rootkit is a toolbox of malicious programs to by-pass kernel protections and take control of a computer. The goal of the
attacker is to obtain root level privileges and hide the attack from the system administrators. “Leave no trace behind,” is
the outdoor code followed by backpackers and campers. Rootkit attacker’s goal is to leave no trace behind without even
going there (to the computer under attack). In 1999, Brumley [15] wrote: “Rootkits are used by intruders to hide and
secure their presence on your system. … To catch a cracker you must understand the tools and techniques he will use to
try to defeat you.” A practical definition of a rootkit is given in {1}.

® Programs and tools to maliciously gain root level access, {1}

Q2 Hide trail by manipulating programs, log files, audit files, etc., and
03 Leave a back door for future remote access.

Proc. of SPIE Vol. 10185 1018503-2

Hannel presented an overview and a best practices approach to dealing with rootkits in 2003 [16]. The average computer
user first learned of rootkits when the Sony music CD copy protection scandal broke in 2005. Russinovich was the first
to report the “copy protection” rootkit in his October 2005 blog [17]. The unthinkable happened next as described in
“Lessons from the Sony CD DRM Episode” [18]. The copy protection [19] actually made it easier for other malware to
infect computers and to add insult to injury, it phoned home to the record label!
It was discovered that the XCP rootkit makes users’ systems more vulnerable to attacks, that both CD DRM schemes install
risky software components without obtaining informed consent from users, that both systems covertly transmit usage
information back to the vendor or the music label, and that none of the protected discs include tools for uninstalling the
software. (For these reasons, both XCP and MediaMax seem to meet the consensus definition of spyware.) These and other
findings outraged many users. [18]

We will review hardware virtualization and hypervisors next. The next battleground for rootkits is hardware virtualization.

3. INTRODUCTION TO VIRTUALIZATION
Modern virtualization concepts can be traced back to Popek and Goldberg’s influential 1973 paper [20]. Singh describes
virtualization as a framework for partitioning computer resources.
Virtualization is a framework or methodology of dividing the resources of a computer into multiple execution environments,
by applying one or more concepts or technologies such as hardware and software partitioning, time-sharing, partial or
complete machine simulation, emulation, quality of service, and many others. [21]

Burger in 2012 describes virtualization as an all-encompassing technology for information technology (IT) industry.
Virtualization technology is possibly the single most important issue in IT and has started a top to bottom overhaul of the
computing industry. The growing awareness of the advantages provided by virtualization technology is brought about by
economic factors of scarce resources, government regulation, and more competition. [22]

Type 1 Hypervisor Type 2 Hypervisor

Application Application
Application Application Software Software
Software Software

Guest OS Guest OS
Guest OS Guest OS
Hypervisor
(Virtual Machine Monitor)
Hypervisor
(Virtual Machine Monitor) Host Operating System

System Hardware System Hardware

Figure 1. Popek and Goldberg Virtual Machine/Hypervisor Types.

Virtualization technology allows a single computer to host multiple virtual machines. Each virtual machine may host an
operating system (called a guest operating system) and application software. Popek and Goldberg define two classes of
virtual machines as illustrated in Figure 1. A Type 1 hypervisor or virtual machine directly connects to the system

Proc. of SPIE Vol. 10185 1018503-3

hardware. The hypervisor includes OS features to interface with the system hardware. The Type 1 hypervisor is commonly
referred to as a bare-metal hypervisor since, the hypervisor directly connects to the system hardware. In a Type 2
hypervisor, a host operating system supports the hypervisor. In both types, the hypervisor provides an interface to each
guest operating system. In doing so, the hypervisor provides each guest operating system with its own “virtual” computer.
Virtual machines are characterized by how virtualization is implemented and if any modifications are required for the
guest operating system [21]. There are also hybrid virtual machines that do not exactly fit the Type 1/Type 2 classification.
In this paper, we will focus on hardware assisted virtualization; where the system hardware provides hardware features to
accelerate operations used by a hypervisor (virtual machine monitor). Hardware assisted virtualization provides the
features required to efficiently implement separation kernels. A separation kernel is an environment where isolated
“sandboxes” are provided for each executing process. An execution sandbox is an isolated computer environment where
untrusted software can be safely executed. Any malicious program behavior will be contained within the sandbox. The
goal is to completely separate and isolate each process so poor programming and malware cannot disrupt any other process.

4. SEPARATION KERNELS AND ROOTKITS

A separation kernel is an environment where isolated “sandboxes” are provided for each executing process. An execution
sandbox is an isolated computer environment where untrusted software can be safely executed. In this construct, any
malicious program behavior will be contained within the sandbox. The goal is to completely isolate each process so poor
programming and malware cannot disrupt any other process. Hardware assisted virtualization provides the features
required to implement separation kernels [23-27] with a small security overhead.
Figure 2 illustrates a separation kernel managing isolated execution environments. The HAP tables provides features for
managing address translation from a virtualized logical address to a global logical address (GLA), and finally to the
machine physical address (MPA) ([real] physical memory address). To isolate a guest operating system from the computer
system. HAP tables allow the hypervisor to create “shadow” page tables for the guest operating system. The shadow table
can be freely updated by the guest operating system; however, the hypervisor maintains the “real” tables and manages the
shadow page tables to isolate the guest OS from the rest of the computer system. Current separation kernel hypervisors
and security hypervisors are utilizing hardware assisted virtualization features to balance computer security, system
performance, and system complexity [25-27]. Hardware assisted virtualization allows security sensitive instructions to be
trapped by the hypervisor. The hypervisor then executes the security sensitive instructions in a controlled environment.
The hypervisor classifies the instruction by operation and computer security. The hypervisor will then run the instruction

Pag
Separation IKCePneO eases lbls
Hardware
Hardware ViP4maOIIa4iow
Features to G^Q¢¢eOera4e
Virtualization
OlbperrNrisoP Fundions Support

Separation Kernel] Separation

Hypervisor
Kernel
creates ¢ooapOe4eOy isoOa4ed
ette¢m4ioru environments

Ho direct interaction, or
communication between
processes is aOOowed

Figure 2. Separation Kernel Example

Proc. of SPIE Vol. 10185 1018503-4

in the appropriate environment: hardware, hardware/software, or emulate it in software. M. Grace, et al. [26], and
X. Xiong, et al. [27] implement hypervisor instruction execution classification. Security hypervisors classify instructions
to maximize hardware execution and minimize emulation. The goal is to provide good security with low overhead.
Rootkit protections fit into two categories: behavior detection and executing prevention. A short list of potential attack
vectors for rootkits includes [28-35]: privilege escalation attack, untrusted kernel extensions, misuse of kernel modules,
control flow attacks, kernel data structure attacks, ye old buffer overflow attacks (old, but still a significant threat [36]),
return to library attacks (return-to-libc), stack manipulation, DMA corruption of kernel memory, and the list goes on ...
What is needed is a computer security mindset for the entire system: hardware, operating system, application software,
software development tools, system test (including security testing), and evaluation. The system lifecycle does not end
after delivery. The system also needs to plan for system upgrades and regression testing.
Figure 3 illustrates a black hat installing a virtual machine rootkit onto a computer system. In 2006, King et al. [37]
demonstrated a virtual machine rootkit subverting Windows. Fannon [24] presents an analysis of the first two virtual
machine rootkits: Blue Pill Virtual Machine Rootkit for both Windows and Linux operating systems [38], and Vitriol
Virtual Machine Rootkit for Apple [39].

Normal System Rootkit Copies installs a hypervisor below

the victim OS and leaves behind malware

T i/ ' 7r Application
i/ Software

Application
Application Software L' Operating
Software U System

Hypervisor
Operating
System
System 1111111
c
System
Hardware Hardware

.
System
Hardware
Under Attack Hacked System

Figure 3. Hypervisor Rootkit Attack Vector

The rootkit infiltrates the computer through any number of potential vulnerabilities. For discussion, we assume the rootkit
has gained root level access and begins to install a hypervisor. The malicious hypervisor then subverts the host operating
system and transfers the host operating system and its system state into a virtualized environment as a “guest” operating
system. At this point the guest operating system is completely isolated from the system hardware and the rootkit installed
hypervisor “owns” (completely controls) the guest operating system and application software. The rootkit’s hypervisor
has full access to all kernel data structures, and “guest” OS code. The rootkit hypervisor can manipulate, alter, and spoof
any “guest” OS system call to the system hardware. Even with state-of-the-art technologies, it is very difficult to detect a
virtual machine or hardware assisted virtual machine rootkit.
Some hardware virtualization based rootkit protection techniques are summarized in Table 1: ① kernel integrity; ② block
rootkit from executing malicious code; ③ enforce control flow as an invariant; and ④ enforce security properties. Fannon
[24] points out the security issues with hardware virtualization are becoming worse: “Although the use of hardware-based
virtualization has been expanding, security mechanisms specific to hardware virtualization have not been keeping pace

Proc. of SPIE Vol. 10185 1018503-5

because of the difficulty of identifying and intercepting malicious instructions before they are passed to the CPU for
execution.”

Table 1. Short Summary of Rootkit Protections

Reference Summary Protection

M. Grace, et al. “Transparent Protection of A hypervisor using hardware virtualization to Enforce Security
Commodity OS Kernels Using Hardware implement a Harvard architecture on x86 Property WX
Virtualization”
X. Xiong, et al. “Practical Protection of Kernel HUKO, a hypervisor integrity protection Kernel integrity
Integrity for Commodity OS from Untrusted with access control policies and security protection
Extensions” aware state transitions.
J. Wei, et al. “Soft-Timer Driven Transient Hypervisor screens all soft timer interrupt Kernel static
Kernel Control Flow Attacks and Defense” requests to prevent rootkit timer attack. analysis
Z. Wang, et al. “Countering Kernel Rootkits with Hypervisor protects kernel hooks using Enforce security
Lightweight Hook Protection” hardware-based page-level protection. property
R. Riley, et al. “Guest-Transparent Prevention of NICKLE virtual machine monitor which Enforce Security
Kernel Rootkits with VMM-based Memory guarantees unauthorized code cannot be Property
Shadowing” executed.
A. Seshadri, et al. “SecVisor: A Tiny Hypervisor Hypervisor allows only approved kernel code Ensures code
to Provide Lifetime Kernel Code Integrity for can be executed integrity
Commodity OSes”
L. Litty, et al. “Hypervisor Support for “Patagonix” hypervisor detects rootkit Detect rootkit
Identifying Covertly Executing Binaries” behavior.

5. SYSTEM SECURITY OBSERVATIONS

Microprocessor architectures always lead the OS developers by several years. Computer security does not have a high
priority with the commodity microprocessor offerings. The security engineer is left with inventing a clever way to achieve
good computer security and minimize overheard. The creative architectures in [28-35] provide clever ways of using
hardware assisted virtualization features for computer security.
A Harvard architecture completely separates instructions and data. A Harvard architecture is much more secure than a
unified (program and data) von Neumann architecture. The x86 architecture is the most well-known von Neumann
architecture. The hypervisor in [26] used hardware assisted virtualization, to emulate with low overhead, a Harvard
architecture on an x86 processor. Context switching presents a large burden on the operating system. We also know Unix
file permission bits are a simple file security mechanism for managing file operations.
The research community needs more interaction between OS developers and hardware developers. Researchers need to
focus on simple hardware features that could drastically reduce the number of lines of software codes. In the OS Friendly
Microprocessor Architecture, we will consider hardware security features based on our observations.

6. OS FRIENDLY MICROPROCESSOR ARCHITECTURE

In order to achieve true separation, all shared system resources must be completely isolated and securely managed. A
good practical example separating control and data is the Post Office example from section 1. In section 5, we presented
some observations about computer security. We know a Harvard architecture is more secure, due to the separation between
instructions and data, than the unified x86 architecture. In Unix, file permission bits create a simple file security
mechanism. The OSFA uses an extended Harvard architecture and implements Unix-like file permission bits for cache

Proc. of SPIE Vol. 10185 1018503-6

bank security headers and memory cell permission bits. The cache bank pipeline architecture of the OSFA also provides
for single cycle context switches.
In Figure 5, the OS Friendly Microprocessor Architecture (OSFA) [40-42] contains 4 cache bank memory pipelines. The
architecture separates instructions and data analogous to a Harvard architecture. The register pipeline and execution
pipeline state cache bank memory pipelines provide for single cycle context switching. The cache bank memory pipeline
(cache bank controller and DMA controller) provides for simultaneous (in parallel) ① process execution and ②
background cache bank memory transfers for context switching.

r r Instruction
Cache Bank
Pipeline
Memory Pipeline Architecture
L J

DMA
ContrIler

Cache Bank

Idle Cache Banks

Cache Bank

Cache Bank

Figure 5. OS Friendly Microprocessor Memory Pipeline Architecture

In Figure 6, the active cache banks are connected to the execution pipeline, inactive cache banks are idle, and swapping
set cache banks are connected to the DMA controller. Swapping set cache banks are copied to upper level caching and
main memory in the background. The register memory pipeline and pipeline state memory pipeline are the cost for the
single cycle context switch. The extra pipelines also provide for more isolation between running processes. The memory
required for both is relatively small. A more detailed description of the cache bank memory pipeline architecture is found
in [40-41].
The key features of the OSFA are the cache bank memory pipeline architecture, cache bank security header, and memory
cell permission bits. The architecture focuses on completely isolating shared resources, and blocking all actions that
exceed the bounds of an execution sandbox. For the OSFA, the basic memory block is a cache bank. Cache bank security
header and memory cell permission bits provide for a hardware level security system. All I/O operations are direct memory
access based on cache banks. The trusted OS is a separation kernel that configures and manages the hardware security
mechanism in the OSFA. The OSFA hardware is the gatekeeper. All authorized operations do not incur any runtime
overhead. Any unauthorized operation (outside the bounds of the hardware sandbox) immediately raises a hardware
exception handled by the trusted OS.

Proc. of SPIE Vol. 10185 1018503-7

 OS Friendly Microprocessor Architecture

Cache Bank
r Instruction Data
I Register r Execution
Pipeline Pipelin Pipeline Pipeline State
Memory Pipeline
Architecture emory emory
and and
4 Memory Pipelines Caches Caches
L

DMA
ontroller

I Cache Bank Cache Bank J

I

dle Cache Banks ó

u CJ
,_...
Cach; _:ik

L
Cache Bark v

t.

Figure 6. OS Friendly Microprocessor Architecture

Trusted
Operating System
(separation kernel)
r, OSFA
Hardware

`` °i 11
Osok4ad SandbottQ2 Cache Bank Cache Bank Cache Bank

NM
7
Untrusted
.10

Untrusted
OS
r OS

Logon Untrusted
Shell Application

Figure 7. Trusted OS and Hardware form Separation Kernel Hypervisor

Proc. of SPIE Vol. 10185 1018503-8

The trusted OS (microkernel) in Figure 7 controls and manages all shared system resources. The trusted OS configures
the security mechanism for the OSFA hardware. When the OSFA hardware detects an attempt to access a resource outside
the process’s sandbox, a hardware exception is raised and handled by the trusted OS. The trusted OS follows this simple
algorithm. A process can request access to a shared resource (memory page, I/O port, etc.). The trusted OS will only
grant access to a shared resource that is not in use by the trusted OS and not currently in use by another process. The
trusted OS sets the security policies and the hardware enforces sandbox bounds. As long as a running process does not
exceed the bounds of its own sandbox, it is allowed to run without any performance penalty from the trusted OS and
OSFA hardware.
Figure 8 shows the cache bank memory format: cache bank security header and memory cell permission bits. The OSFA
hardware only allows the trusted OS to access and modify cache bank security headers and memory cell permission bits.

OS Friendly Microprocessor
Memory Pipeline Architecture [Cache Bank Memory Format

Memory
and Security Header
Caches

Permission
Memory Cell
Bits

Permission
Memory Cell
Bits

Permission
Memory Cell
Bits
Cache Bank

Permission
Memory Cell
Bits
Cache Bank

Cache Bank Cache Bank

Cache Bank Permission

Memory Cell
Bits
Cache Bank

Figure 8. Cache Bank Security Header and Permission Bits

Access Level = Boot = 0

(Hardw =r= m ^a= o^, _.^ =a during boot) Boot Complete Boot Complete
Access Level = 2
Access Level= 3

Trusted OS
Trusted OS
Modules Hypercalls Malloc
DMA
I/O Ports

Figure 9. OSFA Hardware and Trusted OS Boot

Proc. of SPIE Vol. 10185 1018503-9

Figure 9 shows the OSFA hardware and trusted OS access level during boot. During boot, access levels 0, 1, 2, and 3
have their cache bank and memory permission bits configured. The trusted OS may be divided into the core (most trusted
area of the trusted OS) and hypercalls (calls from guest software supporting guest drivers, guest kernel extensions, guest
OS, and applications software.
As illustrated in Figure 10, the separation kernel (OSFA hardware, and trusted OS) create a directory like security
hierarchy. The security scope of visibility extends from level n to lowest branch level. Two level n objects do not have
visibility of each other. There is no visibility across security containers. An object without any sublevels is limited to
visibility of only itself.

Trusted OS
1 , and 2) Trusited OS
ercalls
Sepa a :ion Kerr (Level 3)

kor Kernel
Drivers extensions
Drivers Extensions
Guest OS
(Level 5) (Level 5) (Level 5)
(Level 5) (Level 5)

Guest OS Guest OS
(Level 6) (Level 6)

Application Application Application Application Application

(Level 7) (Level 7) (Level 7) (Level 7) (Level 7)

Isolated Sandbox Isolated Sandbox Isolated Sandbox

Figure 10. Security Hierarchy

7. SIMPLE COMPUTER SECURITY EXAMPLE

We first present a simple example of a running rootkit to introduce a more detailed example in the next section. Figure 11
shows an example OSFA security hierarchy where we assume two rootkits are running. Rootkit_0 is installed at access
level 3. For a rootkit to be installed at a higher privilege level, it would have to undermine the boot process. The OSFA
hardware prevents Rootkit_0 from having access to any other higher access levels: 2, 1, or 0. Rootkit_0 does not
have any sublevel objects. The OSFA and security mechanism limit Rootkit_0 to only resources allowed by the trusted
OS. Any attempt to access a resource outside this hardware sandbox results in a hardware exception handled by the trusted
OS. Rootkit_1 is running inside a sandbox. The possible range of subversion by Rootkit_1 is limited by the security
container (hardware security configuration enforced by the OSFA hardware) as shown. Rootkit_1 could modify the
guest OS kernel data, or spoof any function call from the OS; however, the security container limits the possible range of
subversion to only the security container. Unusual malloc( ) operations and page faults would be a signal to the core
trusted OS of the presence of a potential rootkit. The techniques in [28-35] provide additional tools for detecting a rootkit.
In this example, we are assuming the behavior of the rootkit is limited to the security scope of the sandbox. Any attempt
to access a resource outside the sandbox would instantly raise a hardware exception. Rootkits are malicious and seek total
control; settling for control of a sandbox is unrealistic for a rootkit.

Proc. of SPIE Vol. 10185 1018503-10

 Trusted OS
(1, and 2) Trusted OS Possible Range
Hyperralls
anon Kernel (Lee el 3)
of Subversion

No objects
below Rootkit_O

Drivers Kernel Drivers Kernel Drivers Rootkit_1

Extensions Extensions (Level S)
(Level 5) (Level5) (Level 5)
(Level5) (Level 5)

Guest OS Guest OS Guest OS

(Level 6) (Level 6) (Level 6)

1
r
Application Application Application Application
(Level7) (Level7) (Level 7) (Level 7)

Figure 11. Security Model and Installed Rootkits

8. ROOTKIT AND OSFA SECURITY MECHANISM

Figure 12 shows a rootkit attack from the application software level. When the rootkit attempts to access a resource outside
the sandbox defined by the Trusted OS and OSFA hardware, a hardware level exception is raised and handled by the
trusted OS.

Normal System Under Attack

Application
trd
coftwa

When an resource attempts to

access a resource outside its
sandbox a hardware exception
Trusted OS Trusted OS is raised and handled by the
Hypercalls Hypercalls

i
(Level 3) (Level 3)
trusted OS

Trusted OS

Boot
11111IÌ a - Each Object has a set of
Permission Bits

Figure 12. User Level Rootkit Attack and System Defense

Proc. of SPIE Vol. 10185 1018503-11

In Figure 13, a user level application contains a hidden rootkit. The OSFA hardware only allows the trusted OS to access
cache bank security headers. The security sandboxes and permission bits prevent the rootkit infected application software
from accessing any resource outside its defined sandbox.

Security Header
Access = 1, and 2
Security Header '
Access = 0 Security Header
C

Access = 3

Trusted OS
(1, and 2) Trusted OS
Hypercalls
Separation Kernel (Level 3)
I

Security Header

/
14

Drivers Kernel Guest OS

Access = 4 Extensions
(Level 5) (Level 5)
(Level 5)
Mem = Driver, 4
I

Security Header 4
Guest OS Hardware blocks attacks
(Level 6)
across sandboxes
Access = 5

Mem Access = OS, 5

Application Application
Software (Level 7)
Security Header
Access = 7

Mem Access = App, 7 Isolated Sandbox Isolated Sandbox

Figure 13. Access Levels and Cache Bank Security Headers

An OS function call with a malicious pointer is presented in Figure 14. The malicious pointer, m_ptr, points to a memory
page with a security header showing the Guest OS and Application software do not have access rights to the memory page.
A hardware level exception is raised when m_ptr tries to access the memory page.

Application Guest OS Memory Trusted OS

Exception
Software (Level 6) (Level 2)

Clean up
Application Space

Memory Permission
Page Bits = Not Allowed

Figure 14. OS and Malicious Pointer Function Call

Proc. of SPIE Vol. 10185 1018503-12

In Figure 15, the rootkit in the application software calls the guest OS with a legal malloc( ) function call. Cache bank
security headers define access rights for each cache bank. The memory cell permission bits provide access rights down to
the memory cell level. In [41], a cache bank look up table reduces the number of bits required to store the memory cell
permission bits. The security permission bits for the pointer are set to RWM (Read, Write, and Modify = Not Allowed).
The permission bits allow the pointer to be used as an index pointer to a memory page; however, the application and guest
OS are not allowed to read, write, or modify the address of the pointer. The permission bits allow the pointer to be trusted
by the trusted OS. Any attempt to access the pointer, immediately raises a hardware level exception handled by the trusted
OS.

Security Header
Access = 1, 2, and 3

Application
1
Guest OS Trusted OS
Software
Rootkit Permission Bits =

A Access Pointer Address

= NOT ALLOWED
-A READ, WRITE, MODIFY are
NOT ALLOWED

Memory Permission Memory Permission

Page Bits = Allowed Page Bits = Allowed

t
I i
1

Figure 15. Legal OS malloc( ) Call

In Figure 16, we include a rootkit in the application software. The rootkit’s goal is to obtain administrator level privileges,
and then subvert the OS as illustrated in Figure 3. In a traditional architecture, the OS function calls and memory pointers
present numerous opportunities to exploit a vulnerability and achieve administration level privileges. Figure 16 presents
a high level overview of the OSFA’s security mechanism and system hardware. The rootkit inside an application software
“sheepskin” is blocked by the cache bank security header and memory cell permission bits.
Step 1, Ethernet I/O function call, is to create a pointer to a memory page to store an Ethernet packet. The pointer is
marked Read, Write, and Modify = Not allowed and RegIO is set (pointer to I/O port). The memory page {2} security
header is set to I/O and type = Ethernet. The memory cell permission bits are configured to hold a source address,
destination address, and packet length. These parameters are marked Read, Write, and Modify = Not allowed and they
will configure the Ethernet controller.
Step 2, application sets up parameters and loads Ethernet packet data. To set the Ethernet parameters, the application
program must call the OS and then the trusted OS to set them. The trusted OS checks the parameters to make sure the
parameters are valid. We must verify the parameters are valid before trusting them. The application software cannot read,
write, or modify these parameters. The memory cells for the packet data are configured Write = allowed and Read and
Modify = Not allowed. Write only access {2} provides an additional roadblock for malware attempting to read data
from another running application. The application software copies the packet data into the message block in the memory
page.

Proc. of SPIE Vol. 10185 1018503-13

 Ethernet_I /O Call

Application Guest OS Trusted OS

Software (Level 6) (1, and 2)

Permission Bits =
Access Pointer Address A i READ, WRITE, MODIFY are
A = NOT ALLOWED
NOT ALLOWED

Security Header Ethernet I/O Security Header Ethernet I/O

Source Not allowed Source Not allowed

Destination Not allowed
Destination Not allowed
Length Not allowed Length Not allowed
Message Write allowed Message Write allowed

Configure Ethernet

Application Guest OS Trusted OS

Software i (Level 6) (1, and 2)

Rootkit Permission Bits =

Access Pointer Address A i READ, WRITE, MODIFY are
A = NOT ALLOWED NOT ALLOWED

Ethernet I/O Security Header Ethernet I/O

Not allowed Source Not allowed

Not allowed Destination Not allowed
Not allowed Length Not allowed
Write allowed Message Read /Write

Proc. of SPIE Vol. 10185 1018503-14

Sent Ethernet Packet

Application

Figure 16. Rootkit Attack using I/O Function Call

Guest OS Trusted OS
(Level 6) (1, and 2)
DMA
Software
Rootkit

i
Security Header Ethernet I/O Security Header Ethernet I/O Security Header Ethernet I/O Security Header Ethernet I/O

Source Not allowed Source Not allowed Source Not allowed

Destination Not allowed Destination Not allowed Memory Page Permission Bits Destination Not allowed
Length Not allowed Length Not allowed Length Not allowed
r
Message Writeallowed Message Not allowed Message Not allowed
Step 3, application calls the OS, which calls the trusted OS. The trusted OS marks the message block as access not allowed
by the application software and OS in {3}. The trusted OS then configures the DMA controller and Ethernet controllers
to send the Ethernet packet. Marking the message field access not allowed closes a potential attack vector. Another
process with similar credentials cannot even attempt to write to memory now that the message field is marked access not
allowed.

Security Header Ethernet I/O Security Header Ethernet I/O

==!=
rIM
rt=
Source Not allowed Not allowed
Destination Not allowed
{2} Not allowed
{3}
Length Not allowed Not allowed
Message Not allowed

In Figure 15, Step 3, the rootkit could maliciously modify the pointer. Any attempt to read the address of the pointer will
raise a hardware level exception. As described in [41], buffer overflows, and stack overflow attacks are blocked by
separating instructions and data (extended Harvard architecture), and cache bank security header, and memory cell
permission bits.

9. CONCLUSION
More research into simple hardware features that can significantly reduce OS and software complexity is needed. The
research community needs more interaction between OS developers and hardware developers. A good starting point for
researching hardware and OS trade spaces would be to break down the code in a high assurance OS, like seL4 [12], by
function points. What hardware features could help reduce a significant amount of code? We believe hardware security
features similar to the security mechanism in the OSFA are a good starting point.
The papers by Weaver et al. [4] and Breen et al. [5] described in-band control signaling for the telephone network. Without
a way to authenticate the control signals, anyone could control the 1970’s era telephone network. The computer industry
needs to leave in-band signaling in the 1970’s.
We have provided an introduction to a new architecture with a hardware based security mechanism. The cache bank
security headers, and memory cell permission bits are an extension of Unix file permission bits. The permission bits in
the OSFA create a simple hardware security mechanism. The hardware security mechanism creates sandboxes defined by
the cache banks’ security headers and memory cell permission bits. Any attempt to access a resource outside the sandbox
results in a hardware exception handle by the trusted OS. There is no run time overhead for authorized operations.
We believe the next generation of computer security will incorporate more hardware computer security features.
Hypervisors will use more hardware for security sensitive operations. “Although the use of hardware-based virtualization
has been expanding, security mechanisms specific to hardware virtualization have not been keeping pace …” [24]. We
will need to fully understand all the security tradeoffs.
We are interested in computer security professionals, hardware developers, and OS developers reviewing our security
features for limitations, and performance issues. We are interested in receiving feedback from technology professionals.

10. ACKNOWLEDGEMENT
The authors wishes to thank RDECOM community for the opportunity to develop the OS Friendly Microprocessor
Architecture and improvements. The authors would like to thank our colleagues at AMRDEC for reviewing our paper.
The authors also would like to thank our colleagues at the Army High Performance Computing Research Center at Stanford
University. This work was partially supported through funding provided by the U.S. Army Research Laboratory (ARL)
under contract No. W911NF-07-2-0027.

11. DISCLAIMER AND COPYRIGHT

Reference herein to any specific commercial, private or public products, process, or service by trade name, trademark,
manufacturer, or otherwise, does not constitute or imply its endorsement, recommendation, or favoring by the United
States Government. The views and opinions expressed herein are strictly those of the author(s) and do not represent or
reflect those of the United States Government. Distribution A: approved for public release; distribution is unlimited.
PAO 10603.

Proc. of SPIE Vol. 10185 1018503-15

 12. REFERENCES

[1] Schroeder, M. “Security Kernel Evaluation for Multics (Interim Report),” Report # ESD-TR-75-95, M.I.T - Project
MAC, September 1975.
[2] Wikipedia.org. “Trojan Horse,” 2-3-2017. https://en.wikipedia.org/wiki/Trojan_Horse
[3] Tolkien, J. R. R. The Lord of the Rings, George Allen & Unwin UK, 1954-1955.
[4] Weaver, A. and Newall, N., “In-Band Single Frequency Signaling,” Bell System Technical Journal, 33(6),
1309-1330, (November 1954). https://archive.org/details/bstj33-6-1309
[5] Breen, C. and Dahlbom, C., “Signaling Systems for Control of Telephone Switching,” Bell System Technical Journal,
39(6), 1381-1444 (November 1960). https://archive.org/details/bstj39-6-1381 (January 2016)
[6] Computer History Museum, “Blue Box,” (Dec. 2014). www.computerhistory.org/collections/catalog/102713487
[7] Wikipedia.org. “Evaluation Assurance Level ,” 1-31-2017.
https://en.wikipedia.org/wiki/Evaluation_Assurance_Level#EAL5:_Semiformally_Designed_and_Tested
[8] Schaffer, K. and Voas, J. “What Happened to Formal Methods for Security?” Computer, pp. 70-79, August 2016.
[9] Voas, J. “Insights on Formal Methods in Cybersecurity,” Computer, pp. 102-105, May 2015.
[10] Brosgol, B. “Safety and Security: Certification Issues and Technologies,” CrossTalk: The Journal of Defense
Software Engineering, pp. 9-14, October 2008.
http://static1.1.sqspcdn.com/static/f/702523/9242461/1288742834583/200810-
Brosgol.pdf?token=O%2FzGvGaT3zGAbjAKd9oJcqfMZDI%3D
[11] Harrison, W., et al. “The MILS Architecture for a Secure Global Information Grid,” CrossTalk: The Journal of
Defense Software Engineering, pp. 20-24, October 2005.
http://static1.1.sqspcdn.com/static/f/702523/9277782/1288928922607/200510-
Harrison.pdf?token=jxJuDBgj5pMiQTipExE1AXNNVPQ%3D
[12] Klein, G., et al.: “seL4: formal verification of an OS kernel,” Proceedings of the ACM SIGOPS 22nd symposium on
Operating systems principles, Big Sky, Montana, pp. 207-220, October 11 - 14, 2009.
[13] Keegan, W. “Separation Kernels Enable Rapid Development of Trustworthy Systems,” COTS Journal, pp. 1-3, |
February 2014. www.lynx.com/pdf/Separation_Kernels_Enable_Rapid_Development.pdf
[14] Cohen, D. IE137, April 1990. http://www.ietf.org/rfc/ien/ien137.txt
[15] Brumley, D. “Invisible Intruders: Rootkits in Practice,” Usenix Special Issue on Intrusion Detection, September 1999.
https://www.usenix.org/system/files/login/articles/login_apr15_18_brumley.pdf
[16] J. Hannel: "Linux RootKits For Beginners – From Prevention to Removal,” SANS Institute 2003.
https://www.sans.org/reading-room/whitepapers/linux/linux-rootkits-beginners-prevention-removal-901
[17] M. Russinovich: “Sony, Rootkits and Digital Rights Management Gone Too Far,” October 2005.
https://blogs.technet.microsoft.com/markrussinovich/2005/10/31/sony-rootkits-and-digital-rights-management-gone-too-far/
[18] Halderman J., and Felten E., “Lessons from the Sony CD DRM Episode,” Center for Information Technology Policy,
Department of Computer Science, Princeton University, Feb., 2006.
https://www.copyright.gov/1201/2006/hearings/sonydrm-ext.pdf
[19] Wikipedia.org: “Sony BMG copy protection rootkit scandal,” accessed 1/30/2017.
https://en.wikipedia.org/wiki/Sony_BMG_copy_protection_rootkit_scandal
[20] Popek, G. and R. Goldberg, R. “Formal Requirements for Virtualizable Third Generation Architectures
Communications of the ACM, Vol. 17, No. 7, pp. 412-421, July 1974. http://dl.acm.org/citation.cfm?id=361073
[21] Singh, A. “An introduction to virtualization,” (2004, Jan) www.kernelthread.com/publications/virtualization.
[22] Burger, T. “The Advantages of Using Virtualization Technology in the Enterprise,” Intel Developer Zone, March 5,
2012. https://software.intel.com/en-us/articles/the-advantages-of-using-virtualization-technology-in-the-enterprise
[23] Kulkarni, O., et al.: “Virtualization Technology: A Leading Edge,” International Journal of Computer Application,
Issue 2, Volume 2, pp. 272-287, APRIL 2012. rspublication.com/ijca/april%2012%20pdf/32.pdf
[24] Fannon, R. An analysis of hardware-assisted virtual machine based rootkits, Thesis , Naval Postgraduate School,
June 2014. calhoun.nps.edu/handle/10945/42621
[25] Irvine, C., et. al.: “The Trusted Computing Exemplar Project,” Proceedings of the 5th IEEE Systems, Man and
Cybernetics Information Assurance Workshop, Military Academy, West Point, NY, pp. 109-115 June 2004.
[26] Grace, M., et al.: “Transparent Protection of Commodity OS Kernels Using Hardware Virtualization,” Security and
Privacy in Communication Networks, pp 162-180, 2010.
[27] Xiong, X., et al.: “Practical Protection of Kernel Integrity for Commodity OS from Untrusted Extensions,” NDSS
Symposium, 2011. http://www.internetsociety.org/sites/default/files/xipdf.pdf

Proc. of SPIE Vol. 10185 1018503-16

[28] Wei, J., et al.: “Soft-Timer Driven Transient Kernel Control Flow Attacks and Defense,” IEEE Computer Security
Applications Conference, pp. 97-107, 8-12 Dec. 2008.
[29] Wang, Z., et al.: “Countering Kernel Rootkits with Lightweight Hook Protection,” Proceedings of the 16th ACM
Conference on Computer and Communications Security, Chicago, Illinois, November 9–13, 2009.
https://www.microsoft.com/en-us/research/publication/countering-kernel-rootkits-with-lightweight-hook-protection/
[30] Korkin, I.: "Two Challenges of Stealthy Hypervisors Detection: Time Cheating and Data Fluctuations," Journal of
Digital Forensics, Security and Law, Vol. 10, No. 2, Article 2, pp. 7-38, 2015.
http://commons.erau.edu/jdfsl/vol10/iss2/2
[31] Bratus, S., et al.: “New Directions for Hardware-assisted Trusted Computing Policies,” (Position Paper), Dartmouth
College, NH, August 27, 2008. http://www.cs.dartmouth.edu/~sws/pubs/berlin.pdf
[32] Riley, R., et al.: “Guest-Transparent Prevention of Kernel Rootkits with VMM-based Memory Shadowing,”
https://vsecurity.info, 2008. https://vsecurity.info/pubs/RAID08.pdf
[33] Seshadri, A., et al.: “SecVisor: A Tiny Hypervisor to Provide Lifetime Kernel Code Integrity for Commodity OSes,”
SOSP’07, October 14–17, 2007. www.cs.cmu.edu/~arvinds/pubs/secvisor.pdf
[34] Litty, L., et al.: “Hypervisor Support for Identifying Covertly Executing Binaries,” Security ’08: Proceedings of the
17th USENIX Security Symposium, 2008. [Patagonix]
https://www.usenix.org/legacy/events/sec08/tech/full_papers/litty/litty_html/
[35] Hund, R., et al.: “Return-Oriented Rootkits: Bypassing Kernel Code Integrity Protection Mechanism,” SSYM'09
Proceedings of the 18th conference on USENIX security symposium, pp. 383-398, August 10 - 14, 2009.
https://www.usenix.org/event/sec09/tech/full_papers/hund.pdf
[36] Cowan, C, et al., “Buffer Overflows: Attacks and Defenses for the Vulnerability of the Decade*,” SANS 2000
(System Administration and Network Security) Conference, 1-11, 2000.
http://www.cs.utexas.edu/~shmat/courses/cs380s_fall09/cowan.pdf (February 2014)
[37] King, S. et al.: “SubVirt: implementing malware with virtual machines,” IEEE Symposium on Security and Privacy,
pp. 1-14, 21-24 May 2006.
[38] Rutkowska,J. Blue pill source code 2007. https://bluepillstudy.googlecode.com/svn/trunk/nbp-0.32-public/
[39] Dai Zov, D. A. “Hardware Virtualization Rootkits,” Vitriol, BlackHat Conference 2006.
https://www.blackhat.com/presentations/bh-usa-06/BH-US-06-Zovi.pdf
[40] Jungwirth, P. and La Fratta, P., "OS Friendly Microprocessor Architecture," US Patent 9122610, (Sept. 2015).
[41] Jungwirth, P. and La Fratta, P., "OS Friendly Microprocessor Architecture: Hardware Level Computer Security,”
SPIE Defense+Security Conference, Baltimore, MD, April 2016.
[42] Jungwirth, P. "Computer Security Framework and Hardware Level Computer Security in an Operating System
Friendly Microprocessor Architecture,” US Patent Application April 2017.

Proc. of SPIE Vol. 10185 1018503-17