Operating System Protection Steven M. Swafford University of Maryland University College Prevention and Protection Strategies in Cybersecurity Dr. David T. Bourgeois March 11, 2012



Abstract The idea of hardening an Operating System (OS) is something that has been deliberated over a number of years. At the core of OS security resides what is known as protection rings and these rings address protection in the area of applications, device drivers, and at the center is the kernel. Each ring provides a layer of privilege; the kernel has the highest level of privilege and this privilege decreases as the rings expand outward. The physical act of hardening is to protect the system from intruders, malware, and advanced persistent threats. Each of these vulnerabilities presents distinctive challenges, but at the end of the day hardening an operating system is instrumental to security. The technical aspect of security must be complimented by policies and procedures that clearly define the security methodology of a company. The truth is that security is predicated upon a number of activities and should one of these predicates fail, then the security risk increases. This paper will make a close examination of both the complications and solutions as well as provide various avenues to take in order to provide a reasonable measure of security.



Operating System Protection In order to provide a factual history concerning operating systems in terms of the risks, a close examination of what may occur when the hacker community decides to take action is instrumental in order to both understand and reduce risks. Turning attention back to the time when Microsoft released the Windows 2000 OS, Microsoft challenged the hacker community to infiltrate their OS. In reality, history shows that Microsoft had a lot to learn and that they were not prepared for the results that they were about to experience. It was not days or weeks before the OS suffered a breach; rather it was a matter of hours. Clearly while the operating system may have received bad press, another way of looking at the situation was that Microsoft learned a great deal, around what areas of the OS fell to exploitation. Interestingly enough, LinuxPPC also put this OS up to the challenge and while there were thousands of attacks, LinuxPPC was never breached (Hancock, 1999).

Figure 1. How do breaches occur? As we examine OS security closer, it becomes painfully apparent that there are a great number of options, challenges, and skill sets required. This can be anything from file sharing, process management, memory management, and I/O management (Stallings, 2009). To understand the levels of which hardware and software execute within the OS a fundamental understanding of the layers is required and Figure 2 outlines the scope and



demonstrates at what tier any given process resides. The complexity can range from small to large depending upon the level of confidence desired. The reason for this is solutions may be either software, hardware, or a combination of both. Therefore facing the reality that the OS is the first line of defense, we now can commence planning what activities to pursue. Software For initial consideration, software may be the easiest and an obvious solution when it comes to security. The software does present a measure of protection, but there are also vulnerabilities that come along with any software product. For example, many operating systems come out of the box with utilities such as Remote Desktop Protocol (RDP), Telnet, and File Transfer Protocol (FTP) that may lead to a breach either from internal or external sources. To diminish these risks, each utility must be rationalized and determine if it is desirable or if it can be removed. Removing and uninstalling these unnecessary programs coupled with the installation of antivirus and anti-spyware products and a robust patch management policy will provide a resilient measure of protection. Advanced OS hardening may encompass disk encryption, disabling unused accounts, auditing, and finally the idea of least privilege. Should the OS be hardened to the level that the end user can accomplish a given job without the capability to modify the OS configuration and install unauthorized software; the task of OS security begins to become genuine. Figure 2: Protection Rings



Hardware At the core of the operating system is the kernel and this component of the operating system has the ultimate responsibility when it comes to execution and protection. There are circumstances when applications may require access to the kernel when interfacing with hardware reference Figure 3. This scenario depicts that the kernel will perform its role and either allow or reject the access request based upon the capability of allowing a kernel modification (Jaeger, Van Oorschot, & Wurster, 2011). The idea of embedded systems and built in memory to this day has a great deal of controversy surrounding how this on-board memory is best utilized in terms of security. Wei, Tianzhou, Qingsong, Gang, Nan, Jijun, and Yi (2009) talk to the point of how a microkernel serves an important role by allowing critical functions to be segregated from the actual kernel. The benefits are these instructions execute in a virtual space and therefore provide a higher degree of security in the sense of virus detection and other anomalies. Advantages, Disadvantages, and Scope Much like many other aspects of technology, there are always tradeoffs that must be taken into consideration when it comes to hardening the OS. Every organization must account for their business activities in order to define where aspects of trusted computing, trusted platform module, and trusted computer base reside. The end goal of course is it to minimize the attack surface and when approaching the attack surface it is very important to understand the Figure 3: Hardware driven kernel modifications



pros and cons, services, and utilities (Santana, 2009). To assist in understanding the attack surface Figure 4 outlines previous attacks that outline the greatest risks, these are: 1. Device controls Figure 4: Reducing the attack surface 2. Web controls 3. Application controls

Each level of control serves an important aspect in terms of the technology landscape and business requirements. Software Advantages and Disadvantages Software solutions are typically cost effective and easy to implement. For example, traditional firewalls work well when defining authorized traffic but fell short in the areas of application authorizations and monitoring. The rise of application firewalls address the gap found in software security by using the concept of a whitelist in order to reduce threats around SQL injection, cross site scripting, and other well-known attacks (Rowan, 2007). Consider for a moment the following: 1. Pros: a. Low cost of ownership b. Low risk and quick implementation 2. Cons: a. Even security software solutions come with vulnerabilities b. May require on-site vendor support



Hardware Advantages and Disadvantages Hardware could arguably be the best solution when it comes to security because the complexity between the actual hardware and embedded logic to operate the hardware. For this reason, the idea of using hardware can provide great assistance in security. A key point of difference between software and hardware is the level of isolation in which each interface with the protective rings of the OS (Schroeder & Saltzer, 1972). The drivers for hardware reside inside the ring just above the kernel and therefore provide a much more sophisticated level of security, but for this reason both cost and complexity often becomes a hurdle. Combination of Hardware Advantages and Disadvantages The best possible approach is to combine both the software and hardware solutions, which in most cases will provide the reasonable level of security desired. Here the balance comes into play in terms of cost, ownership, support, and experience levels. Many businesses have an employee base with strong skill sets and support contracts with third party vendors. The combination of these technologies often will yield the desired benefit while at the same time reducing the threat surface while at the same conducting the much needed business activities to be successful. Implementation The first step is to determine which OS to adopt and there are a number of choices available. For example, Windows, OS X, Linux, and many others. Each OS presents its own opportunities and challenges therefore selection may difficult. When it comes to hardening an OS there are many systems that have a smaller attack surface and are typically much more difficult to breach. Of course, the reason for this may be that an OS such as Linux are very easy



to configure in terms of security, but the tradeoff is the ease of use and other business applications required during the day-to-day events. Second and probably the single most important component are a strong patch management process and policy. Stop for a moment and consider the Advanced Persistent Threat (APT), it is not only difficult, but it is also time consuming to stay ahead of the black hat community. Implementing a strong measure of security both on the technology and policy sides of the business is instrumental. Any failure to address both components may lead to a breach in security from either an internal or an external source. Ease and Difficulty Many characteristics influence how simple or how difficult implementation may be. Obviously, the Internet presents the most substantial risk for a variety of reasons. Companies must also take in consideration points that may not be so obvious. For example, the geographical location of a business and the type of work accomplished increases the risk. Assume for a moment that a business providing a localized service in or a small community against a business that provides services to the Department of Defense (DoD). While both face threats, the level of threats clearly differs. Upon closer examination of Figure 5, clearly the United States was the victim to the largest number of breaches in security. Figure 5: Breaches by country



Security Management Practice The idea of tackling security the OS can seem like a problematic task and in truth it can be achieved. Technology changes at a pace that is often difficult to keep up with and because of innovation and modern day business models there is a real concern that security will become a hindrance, which is a valid concern. For any company to be successful in addressing cyber security the foremost step is to implement policy and to provide education for everyone, no exceptions. The logical place to maintain and enforce policies is Human Resources (HR). Within this department, training of both new and existing employees is documented and this documentation serves a number of purposes which may include audits and disciplinary action should the circumstance arise. Policy is instrumental in distinguishing between acceptable and unacceptable behavior. Remember, because employees come from all lifestyles there will be always someone who will either challenge policy verbally or maybe even ignore it all together. Once policies are established, the next step is to communicate the policy to employees. Communications may occur via staff meetings, and training or a combination of both. Challenges and Resolutions The single largest difficulty around security depending upon who is questioned, the answer is often the people. Another important facet of challenges is the lack of information sharing both in the areas of best practices and root cause analysis that come to light after a breach has occurred. An argument could be made that the solution resides within Federal and State regulations. The reality is when it comes to cyber security and breaches many companies fail to report these breaches. This is where regulations can ensure that security vulnerabilities not only are reported



and addressed, but the public can also act should a consumer be directly impacted, and by sharing the details of the breach there is an opportunity to share the details so no other business suffers the same breach. Only after proper research, consideration and collaboration may resolutions quickly begin to rise to the surface, see Figure 6 as an example security framework.
•Define threat surface •Define busisness activities



•Define security practicies •Define risk management strategy

•Define network and application security •Define lifecycle practices Deployment •Audit •Patch Management •Policies and Procedures


Figure 6: Security Framework Prioritize Security The priorities around security can be a moving target because of the ever-changing threat landscape, but there are areas that will remain constant from the threat perspective. For the purposes of this discussion, there are three priorities. First, since the OS is at the center of the cyber world it makes the most sense to do everything possible to remove the high priority vulnerabilities that present the greatest risks. Second, the next level of attention required falls to policies and procedures. This aspect is just as important as the technology itself and maybe even more important.



Finally, is the act of ongoing change management to ensure that the technology and policies do not become outdated. At no point in time does current solutions apply to tomorrow’s problems and the sooner this is understood, the sooner security can adapt to the threat landscape. Summary In conclusion, this paper covered numerous activities surrounding security both in areas of technology and methodologies. At the center of the challenge is the OS and here is where the greatest challenges reside. Hardware and software solution alike each bring their own set of challenges and solutions. In many cases, the hardware solution presents the best bang for the buck when it comes to security often because of the complexity of hardware and how it interacts with the OS kernel. The tradeoff here against software is the financial investment in terms of procurement and support. Finally, is the component of policies and procedures, in order for the technology to be successful people must also have the guidance to operate within the defined boundaries of security.



References Aaraj, N., Raghunathan, A., & Jha, N. K. (2008). Analysis and design of a hardware/software trusted platform module for embedded systems. ACM Transactions on Embedded Computing Systems, 8(1), 8-8-31. Retrieved from Armstrong, I. (2010). Following FISMA. SC Magazine: For IT security professionals (15476693), 21(2), 36-39. Retrieved on March 10, 2012 from Gengxin, S., Fengjing, S., & Sheng, B. (2012). A private data transfer protocol based on a new high secure computer architecture. Journal of Networks, 7(1), 179-186. doi:10.4304/jnw.7.1.179-186 Hancock, B. (1999). Hardening of an OS by putting it out of hack attacks. Computers & Security, 18(6), 463. Retrieved from Jaeger, T., Van Oorschot, P. C., & Wurster, G. (2011). Countering unauthorized code execution on commodity kernels: A survey of common interfaces allowing kernel code modification. Computers & Security, 30(8), 571-579. Retrieved from Kaspersky Lab ZAO (2012). Reducing attacks infographic. Retrieved from Rowan, T. (2007). Application firewalls: filling the void. Network Security, 2007(4), 4-7. doi:10.1016/S1353-4858(07)70043-0 Santana, M. (2009). Chapter 6, Eliminating the Security weakness of Linux and Unix Operating Systems. In Vacca, J. R. (Ed.), Computer and information security handbook. Boston, MA: Morgan Kaufmann Publishers.



Schroeder, M. D., & Saltzer, J. H. (1972). A hardware architecture for implementing protection rings. Communications of the ACM, 15(3), 157-170. Retrieved from Stallings, W. (2009). Operating System Security. In H. Bidgoli (Ed.), Handbook of information security, volume 2. Part 3: Foundations of information, computer and network security. New York, NY: John Wiley & Sons, Inc. Wei, H., Tianzhou, C., Qingsong, S., Gang, W., Nan, Z., Jijun, M., & Yi, L. (2009). A novel operating system on chip with information security support for embedded system. Journal of Software (1796217X), 4(10), 1053-1060. Retrieved from

Sign up to vote on this title
UsefulNot useful