R.C.P.I.T.

, Shirpur

CHAPTER 1

INTRODUCTION
1.1 Why Linux Security is an Issue?
Linux empathized on open source has been developed after first version leased. As network environment got common recently, security policies for that are desperate. Network security policies based on Linux are firewall, Intrusion Detection System, and so on. They show the limit as security policies for the inner server which contains important information. So, the instances of information outflow and invasion happen a lot even though there is a network based security system. The security of Linux OS has built-in functions such as authentication with user identification, reuse prevention of Discretionary Access Control object, and audit trail.

1.2 Linux Weaknesses

Due to weakness of Discretionary Access Control built in Linux it is not safe from hackings like Trojan horse and so on. Therefore, improved policies and related goods to security are required to use Linux as a safer server. Linux is applied to a various fields because it is open source, but it has a weakness of security because of it as well, so it should be solved first. This paper takes a look at kernel-based security solutions so as to solve the weakness of Linux security and defines necessary items of Linux security through this. It checks process control based on unauthorized Linux through improved Linux kernel-leveled security solution and takes a look at improved Linux security solutions. Linux based Unauthorized Process Control

R.C.P.I.T., Shirpur

CHAPTER 2

LINUX SECURITY: BACKGROUND ISSUES

2.1 Linux Background

Linux kernel, high capacity software, performs functions like process management, file system, memory management and network management. Linux kernel source can used freely by anyone and it follows GNU Public License which lets source be modified freely and redistributed. Weakness of Linux is announced in the sites such as Security Focus, CVE(Common Vulnerability and Exposure) and Security Tracker. Requirements of Linux security can be defined as follows Management of user authentication and account Access control on file and directory Management of process Access control on Network Hacking prevention function Self-protection function Performance and installation Recently the number of weakness cases of kernel, a core security mole of Linux, has been increased to about 4 times. It shows that the interest in

Linux based Unauthorized Process Control

R.C.P.I.T., Shirpur kernel has been increased by security related specialists or hackers. It takes a lot of time to analyze the weaknesses of kernel and check. It takes long time to confront by using patches. It is because of some cases: errors from design of Linux kernel, programming errors, and unknown causes. The weaknesses of Linux security are as follows. Weakness patch technology of kernel Access control technology Optional access control Invasion prevention technology Encryption technology Audit record technology Attackers on Linux misuse the weaknesses of security in order to get a system access authority, access to information illegally, change the use of other computer to distribute spam information, and participate in attack on high effectiveness system. Access control of Linux uses an access control method to decide whether the user enables to access to specific resource. Linux based systems use Discretionary Access Control. It controls object access based on user group. The reason why the access control technique causes a problem is that the program intruded by security invasion inherits the access control authority from the user. So, it is not good.

Linux based Unauthorized Process Control

R.C.P.I.T., Shirpur Mandatory Access Control and Role-Based Access Control that use the minimum authority principle are safe. Figure 2.1 shows the process and composition of hierarchical Linux security decision.

Figure 2.1 Layered Linux Security Process

The callback hooks initialized into security loops are defined dynamically as a loadable kernel module but otherwise contain dummy stub functions in the event that no security module is loaded. These stub functions implement the standard Linux DAC policy. The callback hooks exist at all points where object mediation must be provided for security. These include task management program loading, file system management IPC, module hooks, and network hooks.

Linux based Unauthorized Process Control

R.C.P.I.T., Shirpur

2.2 Security
Linux has provided security features which can be traced to the genesis of Unix systems. All Unix based systems have the concept of a super-user generally known as the root. This user has all the administrative powers to manage the system and all the other users generally need the super-users permission to perform actions. This provides a certain sense of security by restricting the scope of actions of a user and therefore limiting the damages that can be caused. Thus, there can only be two types of users on a traditional Unix system, a super user and a normal user. The onus is on the super-user to restrict unauthorized access by normal users. The privileges of overall system access and control is given exclusively to the super user. 2.2.1 Discretionary Access Control (DAC) Discretionary Access Control (DAC) is the traditional approach adopted by most Unix based systems. Security is enforced at the users discretion. The following are some of the highlights of this model: The system consists of a set of users and a set of resources in form of files (or devices). The users of the system belong to one or more user groups. The groups are not mutually exclusive. Every file object is owned by a user. Usually this is the user who creates the file. Each file object is associated with three sets of access rights. These sets are read, write and execute with each having three bits.

Linux based Unauthorized Process Control

R.C.P.I.T., Shirpur The owner of a file can set the access rights for users in his group. The owner of a file can also determine the access rights for other users in the system (not in the owners group). The owner of a file can also set the access rights for himself. With three bits of access rights for each of the above entity, there are totally nine bits associated with a given file object. Practical systems demand a much more flexible method of specifying rules. DAC is not flexible enough to specify complex rules that a security model for an organisation may contain. Also as the name suggests the responsibility of securing information is on the owner or generally the user, of the system. A user can arbitrarily assign or transfer rights on objects to other users, violating the check on information as per the security policy. Hence, the need for MAC. Mandatory Access Control: Mandatory Access Control (MAC) is a set of rules for controlling access based directly on a comparison of the individual clearance (or authorization) for the information and the classification or sensitivity designation of the information being sought, and indirectly on considerations of physical and other environmental factors of control. The mandatory access control rules must accurately reflect the laws, regulations, and general policies from which they are derived. Mandatory access controls also take away the power from the root user as in traditional Unix based systems and subject all users, including administrators, to the access control policies of the system. MAC ensures that security protocols are followed and limit the damage in case of a compromise.

Linux based Unauthorized Process Control

R.C.P.I.T., Shirpur MAC policies are a characteristic of a given organisation. Therefore only the framework is usually provided leaving the actual policy to be stated by an organisation. 2.2.2 Access Control Lists Linux supports Access Control Lists (ACL) where a user can exclusively define access controls for individual users. A user can specify access controls for a specific user or a group of users to any resource he owns. He can also deny access to individual users of a group while granting access to the group as a whole. However, ACL again requires the owner to define access permissions and hence forms a part of the DAC model. 2.2.3 Extended Attributes Extended Attributes (EA) are a specification of the draft POSIX.1e standards. It allows for a file to be tagged with any set of attributes. The attributes are merely key value pairs of data. They are a general mechanism to identify various files and to store application specific meta data about them. EA provides various name spaces to store such data. One such name space called Security can be used to store meta data about security aspects related to that file. This feature is exploited by various security frameworks to specify policies at a very fine grain level. 2.2.4 Linux Security Modules Linux Security Modules (LSM) was adopted into the Linux kernel from version 2.6.8. Any security system requires one to perform access control before the access to the resource is granted. Such access control logic has to be isolated from the normal user space. On Linux systems access controls are

Linux based Unauthorized Process Control

R.C.P.I.T., Shirpur performed by embedding such code within the kernel and invoking the same when a system call is invoked. The access control code hence manages all accesses to a particular resource. LSM isolates the access control logic from the normal user space. It provides a set of callbacks for every available system call on the Linux system. This forces all requests for a given resource to undergo a security check. Any changes to the access control logic is immediately reflected in the next request. A security framework must make use of the callback to implement the access controls necessary.

Figure 2.2 A simple hook for the read system call

2.3 Security Model

The Bell-La Padula model for security introduced the idea of Multi-Level Security (MLS) and the concepts of Objects and Subjects . The Flux Advanced Security Kernel (FLUX) improved the model to implement flexible security policies and a formal architecture for implementing security. The design of

Linux based Unauthorized Process Control

R.C.P.I.T., Shirpur Lothlorien is influenced by both these projects and implements the same in its architecture. The following are the various entities in the Lothlorien Security Model. Subject Any entity capable of making a request for a resource is classified as a subject. Subjects are all the users of the system. Object All resources in the system form the set of objects. An objects consists of files, devices, memory, etc. Role A role is a abstraction of the set of programs that a user is allowed to run. Example: a the role of a user who is able to run a user add, user delete programs may be assigned as an administrator. Levels A logical division of the system. All subjects and objects belong to exactly one level. The logical divisions of the system are required by the TCSEC B1 standards. The logical sections (levels and categories, described latter) represent different levels of security clearance as defined by the security policy. Categories A logical division of a level. All subjects and objects belong to exactly one category. (All subjects and objects belong exactly one level and one category, see Figure 2.3.

Figure 2.3 Levels and categories

Linux based Unauthorized Process Control

R.C.P.I.T., Shirpur

Zone A zone is a pair containing one level and category. Since a given object (or a subject) can only belong to one level and one category, we can say that the particular object (or subject) belongs to a particular zone. Security policy The security policy is a set of rules that maps the set of users to the set of roles. It also specifies the roles with which a user can access the resource in a given zone.

The policy writer maps all programs with a set of suitable roles. Thus, roles are mutually exclusive subsets of a given set of programs in a system. A zone is a Cartesian product of the set of levels and the set of categories. The policy maps the set of subjects to the set of zones (this forms a onto or surjective function). Similarly, the policy maps all the objects in the system to the set of zones (this forms a onto or surjective function). Subjects can be associated with more than one role. However, they can only assume one of the roles while accessing a given resource. The policy writer also specifies which role should a subject assume while requesting access to the specific object. The set of programs that a subject can use is not a union of the roles. The concept of level and category allow the policy writer to specify a fine grained policy thus making the policy flexible. 2.3.1 Example Security Policies Consider a system with three users as administrator, user1 and user2. The users are distributed among the different zones of the system with each user

Linux based Unauthorized Process Control

10

R.C.P.I.T., Shirpur having a set of roles as shown in Table 2.1. The set of roles is maintained in the roles list as shown in Table 2.2. The set of users being mapped to specific roles is maintained in the user role matrix file as shown in Table 2.3.
TABLE 2.1 The entries in the user_zone file

TABLE 2.2 The entries in the roles_list

TABLE 2.3 The entries in the user_role_matrix file

Consider a scenario where user1 attempts to run the mknod task. The algorithm will deny him access as mknod is not listed under any of the roles assigned to user1. He can only read certain files and can run the gcc task. Similarly the administrator cannot run the gcc task as it is not listed in any of the roles assigned to the administrator.

Linux based Unauthorized Process Control

11

R.C.P.I.T., Shirpur

CHAPTER 3

LOADABLE KERNEL MODULE ROOTKIT

3.1 Introduction to Loadable Kernel Module(LKM)
LKM is a kernel program that can be loaded in Linux kernel dynamically or unloaded. Generally, it is used to load a device driver and it is to get rid of discomfort of recompiling the kernel and rebooting whenever a new device is connected to the system. An attacker can load the kernel module that she/he made in the system and control the system freely by misusing these handy functions. This convenience is a very negative aspect for security. Most LKM rootkits use the method which system call function that attacker on normal System Call Interception or hooking made lets be operated. Kernel operates the function that the user wants by referring the address of system call function defined in "sys_call_table", a global variable. LKM rootkits originally makes attackers system call routine operate by changing the location of normal system call function to the address of the system call made by the attacker. LKM Next continues to explain with examples how to snatch system call with some of LKM rootkit module and hide attackers process, file, and specific strings. 3.1.1 Hiding files An attacker has to hide himself/herself(the existence)in the system at any time. Especially, the files or directories an attacker use must not be shown to the system manager. Jobs as follows can be done by using LKM rootkit.

Linux based Unauthorized Process Control

12

R.C.P.I.T., Shirpur "getdents" is the system call called when operating ls instruction, it uses a kernel mode like below and snatches getdents, a system call. That is how the attacker operates hacked_getdentsmade by the attacker. And hacked_getdents routine does files made by the attacker not to output. int hacked_getdents( ) int init_module() orig_getdents = sys_call_table[__NR_getdents]; sys_call_table[_NR_getdents] = hacked_getdents; void cleanup_module() sys_call_table[_NR_getdents] = orig_getdents;

3.1.2 Hiding Strings Traditional Rootkit uses the method that makes attackers process, IP address, or ID not to appear by modifying program like ps, netstat, and who. However, it has a problem that it can be discovered easily by the system manager. When outputting something from the system, it uses write() system call. So it operates write system call in a version of Trojan in order not to output specific strings, i.e. attackers id, ip address, etc. as below by snatching write system call.

Linux based Unauthorized Process Control

13

R.C.P.I.T., Shirpur int hacked_write( ) int init_module( ) orig_write = sys_call_table[__NR_write]; sys_call_table[__NR_write] = hacked_write; void cleanup_module() sys_call_table[__NR_write] = orig_write; The existing Rootkit can hide attackers process, directory, file, and even the fact of connection. However, it provides functions that user wants by changing program codes of user layer like ps, df, netstat, top, and Isof. Hence this Rootkit can be detected easily by checking the size of file, trail of system call used, and integrity of file. So as to analyze a system Rootkit is installed in, no more system instructions can be used to detect, analysis programs like kstat and carbonite. kstat and carbonite are tools based on Linux, so they can be used only in Unix-series OS. Security solutions of kernel module are as follows. LIDS : Linux intrusion Detection System is a host-based intrusion detection system and composed of security management tool and kernel patch to improve kernel security. It provides with the security function of access to file, process, kernel, and network by using ACL.

Linux based Unauthorized Process Control

14

R.C.P.I.T., Shirpur PaX : It is the way of intercepting at the kernel buffer overflow attack using weaknesses of software which happens most recently. Security mechanism is composed of NOEXEC and ASLR techniques mainly. Figure 3.1 below shows the structure of LSM .

Figure 3.1 Structure of LSM Hook

File protection function: kstat p provides detailed information about process. When operating kstat p, process id has to be given as an input value.

Linux based Unauthorized Process Control

15

R.C.P.I.T., Shirpur

3.2 System Access Control

According to development of network environment, openness among systems helps normalize information sharing and provide users with convenience. Whereas it is easier to access to important confidential information of individuals or organizations and system intrusion like hacking has been increasing rapidly as well. The way of encrypting users data and storing it has gathered a big interest. The method to encrypt data and store it is safer than physical security of the system and it can prevent leak of important data from theft of disk itself. Linux based running program is composed of machine codes stored as types of operable in the disk and the set of the codes. When the process related to action is generated, kernel manages information of each process. In order to do scheduling suitable for specific process, kernel will manage Process Control Block, group of information about process control flow and the space for the scheduling unit. The information that kernel manages for each process includes that a process uses several resources if being operated and it gets back File Descriptor as a result. Also instructions and data signals being operated in the process which is belonged to a tree related to process have to be managed independently for each process. Kernel allocates a lot of data structures to manage information of process when process is allocated. The structures in core position among data structures for process are task_struct and file_struct.

Linux based Unauthorized Process Control

16

R.C.P.I.T., Shirpur

Figure 3.2 Flow of Process

In order to store processor list in kernel module, the information for the process is generated in array. Initial array is applied after initializing. After achieving a process list, it sets an array through process ID and checks the elements of the list. If there are any differences between lists of process, it regards there is a process and decides it as an abnormal access. It takes a stage that follows the information of the process that kernel progresses.

Linux based Unauthorized Process Control

17

R.C.P.I.T., Shirpur Inode object manages the data block of process being existed in the actual disk. Proposed module operates check on process and forms log files, and then sends log file and the message of it. Figure 3.3 shows the information of process achieved form instructions and Figure 3.4 is a checking result of information of the process by circulating linked list of tack-struct.

Figure 3.3 Information of Process(1)

Linux based Unauthorized Process Control

18

R.C.P.I.T., Shirpur

Figure 3.4 Information of Process(2)

Log file used in system operation in Figure 3.4 below is necessary information of system operation to analyze intrusions. When LKM detection module finds LKM rootkit, the module has to sent the current log file and system information to backup system before an attacker erases the traces of intrusion. That is how to provide an integrity. Log files are the record that records users' behaviors accessed to the system. Therefore, they have semantic information of security like what the intruder from outside did in the system and what instructions users used and system operating information like what the system managed and errors.

Linux based Unauthorized Process Control

19

R.C.P.I.T., Shirpur

Figure 3.5 Backup Logfile

The intruder that achieved the root authority enables to delete log files or change. Once the job has been done well, anything for the improvement of the system security cannot be done because there will not be any information about intrusion path or intrusion method because the related data to trail or analysis has been deleted or modulated even if the manager gets to know the fact of intrusion later. Hence the gain of logs is a basis of managemental security. In the first step among 3 steps of collecting information of process, the LKM Rootkit detection module proposed earlier transfers the log information of the system being operated currently when detecting LKM Rootkit.

Linux based Unauthorized Process Control

20

R.C.P.I.T., Shirpur

These logs are the important intrusion information to be used for the intruder to erase her/his traces has to be backup in the state of integrity.

Linux based Unauthorized Process Control

21