Intrusion Detection and Prevention Systems

Alberto Grand

Contents
Introduction 1 Overview 1.1 Purposes of IDPSs . . . . . . . . . . 1.2 Accuracy of IDPSs . . . . . . . . . . 1.3 Classification . . . . . . . . . . . . . 1.4 Common detection methodologies . . 1.4.1 Signature-based detection . . 1.4.2 Anomaly-based detection . . 1.4.3 Stateful-protocol analysis . . 1.5 IDPS architectures and components 1.6 Evasion techniques . . . . . . . . . . 2 3 3 3 4 4 4 5 6 6 7 10 10 10 12 13 13 14 14 14 15 16 16 16 17 17 17 18 18 19 19 19 21 21 22 22 23 23 23 25 1

. . . . . . . . .

. . . . . . . . .

. . . . . . . . .

. . . . . . . . .

. . . . . . . . .

. . . . . . . . .

. . . . . . . . .

. . . . . . . . .

. . . . . . . . .

. . . . . . . . .

. . . . . . . . .

. . . . . . . . .

. . . . . . . . .

. . . . . . . . .

. . . . . . . . .

. . . . . . . . .

. . . . . . . . .

. . . . . . . . .

. . . . . . . . .

. . . . . . . . .

. . . . . . . . .

. . . . . . . . .

. . . . . . . . .

. . . . . . . . .

. . . . . . . . .

. . . . . . . . .

. . . . . . . . .

. . . . . . . . .

2 Types of IDPS technologies 2.1 Network-based IDPS . . . . . . . . . . . . . . 2.1.1 Components and architectures . . . . 2.1.2 Detection capabilities . . . . . . . . . 2.1.3 Prevention capabilities . . . . . . . . . 2.1.4 Detection accuracy and tuning . . . . 2.1.5 Limitations . . . . . . . . . . . . . . . 2.2 Wireless IDPS . . . . . . . . . . . . . . . . . 2.2.1 Components and architectures . . . . 2.2.2 Detection capabilities . . . . . . . . . 2.2.3 Prevention capabilities . . . . . . . . . 2.2.4 Detection accuracy and tuning . . . . 2.2.5 Limitations . . . . . . . . . . . . . . . 2.3 Network Behavior Analysis (NBA) system . . 2.3.1 Components and architecture . . . . . 2.3.2 Detection and prevention capabilities . 2.3.3 Detection accuracy and tuning . . . . 2.3.4 Limitations . . . . . . . . . . . . . . . 2.4 Host-based IDPS . . . . . . . . . . . . . . . . 2.4.1 Components and architecture . . . . . 2.4.2 Host agents architecture . . . . . . . . 2.4.3 Detection and prevention capabilities . 2.4.4 Detection accuracy and tuning . . . . 2.4.5 Limitations . . . . . . . . . . . . . . . 2.5 Honey pots . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . . . . . .

3 Snort: an open-source NIDPS 3.1 Modes of operation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3.2 Components . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . Conclusions

Introduction
Intrusion is one of the most widely known threats to security in computer and network systems and has existed as long as the systems themselves. Back in the 1980s, James Anderson published a paper in which he pointed out that audit trails contained vital information that could be valuable in tracking misuse and understanding user behaviour, thereby laying the foundations of modern intrusion detection concepts. He defined an intrusion as a possibly deliberate unauthorized attempt to access information, manipulate information or render a system unreliable or unusable. An ideal, perfectly secure system should not allow intrusion to occur; in practice, no real system being completely secure, intrusion attempts can succeed. It is therefore important to detect them and limit as much as possible the damage they can produce. Several intrusion detection techniques have been devised for that purpose. The word intrusion detection refers to the process of monitoring all activity occurring in a network or a computer system searching for signs of possible incidents, which are violations, or threats of violation, of computer security policies, acceptable use policies or standard security practices. Such incidents may have different origins. They may be intentionally produced by malicious users outside a local network who are trying to gain unauthorized access to a system through the Internet, to prevent a system from providing a given service or to cause unnoticed information leakage from a system. They might also originate from authorized users of a system misusing their privileges or trying to gain additional privileges. However, studies have shown that a large amount of computer security incidents are caused by insiders that are inadvertently violating a security policy. Benign intruders, such as a user exploring an unauthorized website, might not be particularly harmful and could be tolerable, although they do consume resources and slow performance for legitimate users. Nonetheless, there is no way to know in advance whether an intruder will be benign or malign; as a consequence, there is a motivation for controlling this problem. Some of the attacks coming from the first category of users may still be countered by using a firewall, yet not all: some attacks may not violate any specific firewall policy or rule and therefore be allowed. What is more, a firewall becomes completely useless when attacks, whether intentional or not, are coming from insiders; hence the need for monitoring and analyzing the events occurring in a given system and deploying real-time solutions when an incident is detected. An Intrusion Detection System (IDS) is a piece of software or an appliance that automates the intrusion detection process. An Intrusion Prevention System (IPS) offers all the functionalities of an IDS, but it additionally attempts to stop possible incidents. IPS technology is considered by some to be an extension of IDS technology; NIST’s special publication [1] concerning IDS and IPS technology resorts to the acronym IDPS (Intrusion Detection and Prevention System) which, although not in widespread use in the security community, well accounts for the tight connection between these two technologies.

2

Chapter 1

Overview
1.1 Purposes of IDPSs

The primary purpose of an IDPS is the detection of possible incidents. For example, an IDPS could produce logs of the activity on a given network, identify when an attack is being deployed, implement the appropriate countermeasures and report the incident to security administrators. Many IDPS can also be configured to recognize violations of an organisation’s security and acceptable use policies, for example transfers of inappropriate material throughout the company’s network or downloads of large databases onto laptops. Some IDPS are also able to identify reconnaissance activity, which may indicate that an attack is imminent. For example, some forms of malware may perform port scans in order to identify possible targets for an attack. An IDPS might be able to block reconnaissance activity and notify security administrators, who may then enable other security controls and counteract the attack. In addition to identifying security incidents and supporting incident response efforts, IDPS technology may serve a number of other purposes: • Identifying security policy problems. An IDPS could be configured with a set of firewalllike rules and perform checks on the traffic, notifying the administrators when traffic that should have been blocked by the firewall but was not, probably due to misconfiguration, is detected. • Documenting the existing threats to an organisation. Intrusion detection enables the collection of information that proves useful in understanding intrusion techniques and the frequency and characteristics of the threats to which an organisation is exposed and in tailoring the security measures accordingly. • Deterring individuals from violating security policies. Being aware that their activity is being monitored, individuals might be less prone to violate the organisation’s policies out of fear of being detected.

1.2

Accuracy of IDPSs

A feature common to all IDPS technologies is that they cannot assure completely accurate detection of threats. Two types of mistakes can occur: false positives and false negatives. The expression false positive, which is in use in a variety of other fields, besides computer security, indicates a situation in which a test (e.g. intrusion detection, anti-virus scan, HIV test) turns out to be erroneously positive. Conversely, a false negative occurs when a test fails to detect suspicious activity (or a virus, a disease etc.). Figure 1.1 illustrates, in very abstract terms, the typical behaviour of an authorized user (right curve) and of an intruder (left curve). As suggested by the partial overlap between the two curves, it is not possible to eliminate all false positives and negatives; in most cases, reducing the occurrences of one increases the occurrences of the other. Sometimes organizations choose to reduce the number of false positives, so that more potentially malicious events are detected. This way, deeper analysis 3

is necessary to discern true malicious events from false positives. This may seem harmless enough; however, high rates of false positives can damage productivity due to temporary system down-times and require a lot of resources. Altering the configuration of an IDPS to improve its accuracy is known as tuning.

Figure 1.1: Authorized user behaviour and intruder behaviour profiles.

1.3

Classification

IDPS technologies may be classified according to different parameters, namely: • the methodology(ies) they employ to detect intrusions: signature-based detection, anomalybased detection and stateful protocol analysis. Common detection methodologies will be presented in section 1.4. • the functionalities they provide, which ultimately differentiate passive systems (IDSs) from reactive systems (IPSs). • the type of events they monitor, which are closely related to the type of systems they guard: a wired network, a wireless network or a single host. In addition to these, a fourth type of IDPS may be identified, which is known as Network Behaviour Analysis (NBA) IDPS. Types of IDPSs will be discussed in chapter 2.

1.4

Common detection methodologies

The primary classes of detection methodologies are signature-based detection, anomaly-based detection and stateful protocol analysis. Many IDPS technologies integrate multiple techniques to provide more accurate detection. An overview of the three methodologies, along with their major drawbacks and limitations, is given in sections 1.4.1 through 1.4.3.

1.4.1

Signature-based detection

A signature is a pattern which characterises a known threat. Examples of signatures include the following: a Telnet attempt with a username of root, which is a violation of an organization’s security policy; a series of TCP SYN packets sent to many different ports in rapid succession, which may

4

indicate that a port scan is being performed; an e-mail with a subject of “Free ringtones” and an attachment filename of “ringtones.exe”, which are characteristics of known forms of malware. Signature-based detection lies in comparing observed events against a set of known signatures to identify possible incidents. Much like an anti-virus scanner, detection capabilities depend on receiving regular signature updates, to keep in touch with variations in hacker technique and new types of threats as they are developed. While being very good at detecting known threats, this technique is largely ineffective at detecting previously unknown threats. What is more, signature-based solutions are relatively easy to deceive by slightly varying the attack pattern. For instance, if an attacker changed the attachment filename in the previous example to “ringtones2.exe”, a signature looking for “ringtones.exe” would not match it. There are other problems that affect the accuracy of signature-based detection. Firstly, a comprehensive signature database, while providing protection against a broader range of threats, inevitably increases the computational load for the system charged with analyzing each signature. When the amount of traffic to be searched becomes substantial, some packets may have to be dropped, thus potentially overlooking some threats. To avoid this, traffic feeds may have to be split, analysed by different systems and recombined after analysis, increasing complexity and cost. Secondly, signature-based techniques only compare the current unit of activity, such as a packet or a log entry, to a list of signatures and therefore have little understanding of many network and application protocols. They cannot track the state of complex communications or pair a request with the corresponding response. This limitation prevents signature-based detection methods from identifying attacks that comprise multiple events if none of them individually contains a clear indication of an attack. Finally, threats disguised by means of evasion techniques are usually not detected by signaturebased methodologies. Evasion aims at modifying the attack in such a way that its appearance is different, but its effects are the same. For example, an attacker could encode text characters in a particular way, knowing that the destination host understands the encoding, whereas the IDPS might not. Section 1.6 includes examples of common evasion techniques.

1.4.2

Anomaly-based detection

The foundations of this detection technique lie in comparing the events observed on a network or a host with what is considered to be normal activity for them, in order to identify significant deviations. Two approaches are possible: threshold detection and profile detection. The first, more simplistic one involves defining thresholds, independent of users, for the frequency of occurrence of various events. Occurrences of specific events are then counted over an interval of time; if the count surpasses what is considered a reasonable number, then intrusion is assumed. Threshold analysis by itself is a largely inaccurate technique; thresholds and time intervals must be fixed, but because of the variability across users, hosts etc. Such thresholds are likely to generate a lot of false positives or a lot of false negatives. Profile-based detection entails the use of profiles that represent the normal behaviour of such entities as host, users, networks or applications. Statistical methods are then used to identify significant deviations from profile-recorded behaviour. Profiles are developed by monitoring the characteristics of typical activity and take into account a number of attributes (number of e-mails sent by a user, number of failed login attempts for a host, level of processor usage, percentage of available bandwidth used for web activity during workday hours, etc.). An initial profile is generated over a period of time (typically days, sometimes weeks) sometimes called a training period. Profiles can be either static or dynamic. Once generated, a static profile is unchanged unless the IDPS is told otherwise; because typical activity can change, static profiles eventually become obsolete and therefore need to be regenerated periodically. On the other hand, dynamic profiles are adjusted on a regular basis as additional events are observed, so they cannot become outdated. However, their adaptation capabilities make them susceptible to evasion attempts from attackers. For example, an attacker could perform small amounts of malicious activity once in a while and gradually increase the frequency and quantity of such activity over time. If the change is sufficiently slow, the IDPS might regard malicious activity as normal behaviour and include it in its profile. Inadvertently including malicious activity as part of a profile might also occur 5

during the training period, when the initial profile is built. In some cases administrators can manually remove parts of a profile that are known to represent malicious activity. Another problem with building accurate profiles is that, computing activity being very complex, it is virtually impossible for an IDPS to observe all instances of legitimate behaviour. As a result, some events which deviate from the recorded profile may trigger alerts and thus generate false positives. Furthermore, due to the complexity and large number of logged events, it could be difficult for analysts to understand why a particular alert was generated and validate that it is accurate and not a false positive.

1.4.3

Stateful-protocol analysis

Stateful-protocol analysis is to some extent similar to anomaly-based detection. It differs from the latter in that it does not use network- or host-specific profiles generated by the IDPS itself; instead, stateful-protocol analysis relies on vendor-developed universal profiles that specify how particular protocols should and should not be used. Profiles enable the IDPS to understand and track the state of network, transport and application protocols that have a notion of state. It can thus identify unexpected sequences of commands, such as issuing the same command repeatedly or issuing a command without first issuing another command upon which it is dependent. For protocols that require authentication, the IDPS can keep track of the authenticator used for each session and record an authenticator used for suspicious activity, which is helpful when investigating an accident. Some IDPSs are also capable of distinguishing among multiple users or classes of users to define acceptable activity differently. Stateful-protocol analysis could for example keep track of a File Transfer Protocol (FTP) session. When the session is started, it is initially in the unauthenticated state. Unauthenticated users should only perform a limited range of commands; unexpected commands might be a symptom of malicious activity. An important part to understanding state is pairing requests with responses; so, for example, when an FTP authentication attempt occurs, the IDPS can determine if it was successful from the status code found in the corresponding response. Protocol analysis includes reasonableness criteria for individual commands, such as minimum and maximum lengths for arguments and the presence of binary data within argument strings. Stateful-protocol analysis methods use protocol models which are typically based on protocol standards from software vendors and standard bodies, such as IETF or RFC. They usually also take into account variances among implementations. However, many standards are not exhaustively complete in explaining some minor details of the protocol; in addition, many vendors either violate standards or add proprietary features which override standard features and complete details are often unavailable. As a result, an IDPS may find it extremely difficult to perform accurate, comprehensive analysis or its interpretation of a protocol may conflict with the way it is implemented in some applications or operating systems on the end host. Other drawbacks to stateful-protocol analysis are that it is very resource-intensive and it cannot detect attacks that do not violate the characteristics of generally acceptable protocol behaviour.

1.5

IDPS architectures and components

All four types of IDPS solutions share the same basic components, although not all of them are fundamental in all cases. Major components include the following: • Sensors or agents. They are in charge of monitoring and analyzing activity. The word sensor normally applies to IDPSs that monitor networks, while agent is typically used for host-based IDPSs. In the former case sensors could be software programmes, but they are usually ad-hoc appliances, whereas in the latter case an agent is a piece of software running on the host it guards. • Management servers. They are centralized devices that manage the sensors or agents, receive information collected by the sensors and process it. Some types of sensors or agents are able

6

to perform basic analysis of the events they detect; management servers, on the other hand, can perform several kinds of analysis and correlate events by matching information coming from different sensors. Management servers are available both as appliances and software-only products. Small IDPSs solutions may not need a management server, but most do include one; larger-scale deployments usually have multiple management servers and in some cases there are two tiers of them. • Database server. It is used as a repository for event information recorded by the sensors or agents or processed by the management servers. It is not a vital part of an IDPS, although many IDPS solutions support it. • Console(s). A console is a software programme that provides an interface for the IDPS’s users and administrators. It is typically installed on a desktop or laptop computer and can provide administration-only functionalities, such as configuring sensors or agents and applying software updates, or both administration and monitoring capabilities. User consoles typically provide monitoring-only capabilities. IDPS components may be interconnected in a number of ways, which provide different trade-offs between level of security and complexity of implementation. The simplest way to interconnect IDPS components is through the organisation’s standard internal networks. No additional wires or devices (besides the IDPS itself) need to be set up. This solution is easy and inexpensive, but it is the least secure of all for a number of reasons. Firstly, IDPS components need to communicate and they do it using the same network they are guarding; as a consequence, they must have an address on that network, which reveals their presence to the rest of the network and makes them possible targets of an attack. Secondly, communication between IDPS components should be relatively fast and reliable, but there is no such assurance if internal traffic is intense or under adverse conditions; for example, if the organisation is being subject to a DDoS or worm attack, there might not be sufficient bandwidth for the IDPS to function properly, which ultimately makes it useless. A better solution is the use of a virtual local area network (VLAN) for IDPS management. The virtual management network still relies on the organisation’s standard network, but it is logically separated from the latter and virtually invisible to other external users. However, misconfiguration of the VLAN may lead to the exposure of IDPS data. Furthermore, as with the first solution, incidents that saturate the organisation’s primary networks still negatively impact the availability and performance of the IDPS. A separate management network, strictly designed for security equipment management, provides an optimal solution. When a management network is used, each sensor has an additional network interface, known as a management interface, that connects to the management network; the sensor is unable to pass any traffic between the two interfaces. This configuration is also known as stealth mode. Other components, such as the management server and the database server, are attached to the management network only. This architecture effectively isolates the management network from the production network. IDPS components can operate in a transparent way, without the need for an IP address on the primary network and can thus conceal their existence from attackers. In case of an attack, adequate bandwidth is available for the IDPS components to function appropriately. Disadvantages to this solution include the additional costs in networking equipment, since a special network is devoted to IDPS interconnections, and the inconvenience, for administrators and users, of having to use separate computers for IDPS management and monitoring and normal production activity (using PCs with double network interfaces that connect to both the internal and the management networks is, of course, forbidden).

1.6

Evasion techniques

Soon after IDPS technology emerged in the corporate environment, hackers introduced several methods for eluding detection. At first, evasion techniques were simple (DoS attacks, false positives and patternmatching techniques), but over time they became increasingly complex and subtle (session splicing, 7

fragmentation and polymorphic shellcode). The rest of the section will briefly outline some evasion techniques, both simple and more complex ones. Pattern-matching weaknesses. As previously mentioned, signature-based detection techniques are exposed to known attacks in which the distinguishing marks have been slightly changed, so as not to trigger an alarm in the IDPS. It is difficult to develop effective signatures that match such obfuscations. For example, if an IDPS looks for the string GET /cgi-bin/phf?, the string GET /cgi-bin/aaaaaaaa/..%25%2fp%68f? will not be detected as harmful, but the end host will process it correctly. Most modern IDPSs have improved in this area, although weaknesses still exists. Unicode evasion techniques. Unicode is an industry standard which allows computers to consistently represent and manipulate text in most of the world’s writing systems (some 100,000 characters overall). It includes several encodings, among which the widespread UTF-8 standard, which maximizes compatibility with the ASCII representation. The standard has been implemented in many recent technologies, including XML, the Java programming language, the Microsoft .NET Framework and modern operating systems. Because of some ambiguities inherent in character representations, Unicode could be exploited to deploy attacks. In 2000 Bruce Schneier pointed out that, with the Unicode character set, multiple representations of a single character were possible. This is because UTF-8 went through several extensions, and each time the representation was extended a byte, the complete previous set of characters was remapped. Thus, for example, when UTF-8 started three-byte representations, it repeated the Unicode code points from the two-byte representation. In addition, there are code points that may be used to modify the previous code point. In a security context it is important to understand the meaning of a character and these problems make the task more difficult, especially for IDPS systems. Although the Unicode Consortium revised the Standard and made multiple representations illegal, some applications may still be using the old standard, so IDPS signatures should still include all possible variants. A clear example of the kind of problems that arose out of multiple representations is the Microsoft IIS 4.0 / 5.0 Extended Unicode Directory Traversal Vulnerability. IIS checked for directory traversal before decoding UTF-8 and thus missed UTF-8-encoded directory traversals, but the operating system did not. As a result, an attempt to escape out of web root by submitting an URL like http://myvictim/../../winnt/system32/cmd.exe was detected by IIS and an error message was generated; but if, instead, the ../ part was UTF-8-encoded, IIS would not understand it and the attacker could then run other programs and access other files. IDPS vendors released new signatures for this kind of attack, but many of them would only catch a few Unicode variations of the string ../, leaving the way open to other variants of the same attack. Denial of service (DoS) and false positives. Many IDPSs have central logging servers that are used solely to store IDPS alert logs. If attackers are able to find out the central logging server’s IP address, they could slow it down or even crash it using a DoS attack. Once the server is shut down, attacks could go unnoticed because alert data is no longer being logged. This problem can be solved by isolating the network that IDPS components use to communicate from the organisation’s internal network. Another way to prevent logging servers from recording alert data and enable attacks to go unnoticed is to fill their disk space by sending false positives. This technique could also be exploited to generate a lot of “log noise”, by forcing IDPSs to generate a large number of false reports. This could make the task of differentiating between false positives and real attacks very challenging. Session splicing. The idea behind this technique is to split data between several packets, making sure that no single packet matches any IDPS signature. It exploits how some IDPS do not reassemble packets before performing pattern matching on data or stop reassembling after a certain delay has elapsed between two consecutive packets belonging to the same stream. Many modern IDPSs can nowadays handle stream reassembling, substantially putting an end to session splicing.

8

Fragmentation. Fragmentation attacks are similar to session splicing, in that attackers send packets which do not individually trigger any alarm. However, fragmentation is generally more powerful than session splicing. Two methods are commonly used to evade IDPS detection. The first method overwrites part of a previous fragment, while the second one replaces a complete fragment. This enables attackers to write entire packets of garbage, which are harmless from an IDPS’s point of view, and craft their attack to blend in with standard protocols. Some IDPSs have found a way to handle these types of attacks through reassembly; however, this may require in-depth knowledge of the operating system in use on the target host and of how it implements the TCP/IP stack. Time-To-Live attacks. Yet another way to bypass intrusion detection, time-to-live attacks require a somewhat accurate knowledge of the internal network topology. If the attackers know the distance (in terms of hops) to the end host, they could send packets which would only reach the IDPS sensor by setting small TTL values, while packets with larger TTL values would be sure to reach the end host. This would enable to inject garbage packets in the stream received by the IDPS and confuse it, while the end host would still receive a clean packet stream. This technique can be countered by changing the TTL flag to larger values in every received packet. If the IDPS additionally collects and monitors TTL information for each host in the local network, further information on possible attacks could be inferred. Invalid RST packets. The TCP protocol uses checksums to ensure that communication is reliable. Every time a packet is received, the end host computes the checksum and compares it to the received one; if they differ, the packet is dropped. Attackers may exploit this feature to send RST packets with an invalid checksum. The end host will verify the checksum value and discard the packet, but some IDPSs may interpret the packet as an actual termination of the communication and stop reassembling packets. However, the end host would continue accepting packets from the attacker, confusing the IDPS. Polymorphic and ASCII shellcode. A shellcode is a relocatable piece of machine code used as a payload in the exploitation of a software bug. Since intrusion detection can detect signatures of simple shellcode, the latter is often disguised through simple encoding or encryption and contains a stub that decodes the shellcode that follows. This is known as polymorphic shellcode and hides the commonly used strings within shellcode, which can be completely different each time it is sent, making shellcode signatures useless. ASCII shellcode is similar to polymorphic shellcode; it contains only valid ASCII characters, which allows attackers to bypass enforced character restrictions within input strings. Strings are hidden within the ASCII shellcode, making pattern matching very difficult. Using ASCII for shellcode is very restrictive, because not all Assembly instructions convert directly to valid ASCII values. This restriction can be bypassed using other instructions or a combination of instructions which map to ASCII and serve the same purpose.

9

Chapter 2

Types of IDPS technologies
The four main types of IDPS technologies – network-based, wireless, NBA and host-based – will be discussed in sections 2.1 through 2.4. Honey pots, which are not IDPS technologies, but can nonetheless be associated with them, are briefly outlined in section 2.5.

2.1

Network-based IDPS

A network-based IDPS (NIDPS) monitors wired network traffic for particular network segments or devices and analyzes network, transport and application protocols to identify suspicious activity. Most of the analysis is done at the application layer. Transport and network layers are also analyzed to identify attacks at those layers and to help analysis at the application layer (e.g., a TCP port number could indicate which application is being used). The hardware-layer is sometimes analyzed to a limited extent.

2.1.1

Components and architectures

A typical network-based IDPS includes all the basic components of an IDPS. Whenever feasible, these components should be connected through a management network. The sensors to be used with NIDPS solutions are equipped with network interfaces placed into promiscuous mode, which allows them to accept all packets, regardless of their intended destination. Sensors are available in two formats: appliance and software-only. An appliance-based sensor is a piece of specialized hardware which comprises NICs optimized for efficient capture of traffic; it also includes specialized processors or other hardware components that assist in analysis, as well as sensor software, which might reside in firmware for increased efficiency. It often uses a special operating system that administrators are not supposed to access directly. A software-only sensor is a piece of software which is intended for use with normal hosts, provided that they meet certain requirements. It can be either in the form of a customized OS or a normal application that runs onto a standard OS. Sensors can be deployed in one of two modes: inline or passive. This choice influences the prevention capabilities of the IDPS solution, as shall be discussed in section 2.1.3. Inline sensors. An inline sensor is located so that the network traffic it is intended to monitor must pass through it. The main reason for deploying inline sensors is to enable them to stop attacks by blocking traffic. They are usually placed where firewalls and other network security devices would be placed, i.e. at the boundaries between networks. Some inline sensors maybe integrated into an existing firewall device. If this is not the case, inline sensors are preferably located on the more secure side of a network division; for example, if a firewall is present at the boundary between an external and an internal network, the sensors should be placed on the internal side, so that they have less traffic to process. Depending on the circumstances, they might also be placed on the less secure side of the division to provide protection for and reduce the load on the dividing device. Figure 2.1 shows an inline sensor deployment.

10

Figure 2.1: Network-based IDPS sensors: an inline deployment. Passive sensors. A passive sensor, on the other hand, is deployed in such a way as to monitor a copy of the network traffic, but no traffic actually passes through it. Passive sensors may be used to monitor key network segments, such as activity on a demilitarized zone (DMZ) subnet. A typical deployment is shown in Figure 2.2. There are several means to passively monitor traffic; the main ones are spanning ports, hubs, network taps and IDS load balancers. A spanning port, also known as SPAN (Switch Port ANalyzer) port, is commonly available on switches and allows to see all traffic that passes through the switch. It works by configuring the switch to copy trasmitted traffic or received traffic (or both) from one port or VLAN to another port. It is easy and inexpensive to use and, since most switches support spanning ports, it does not require additional hardare; also, it does not impact the functioning of other devices, such as firewalls. However, it can lead to some problems. If the switch is not correctly configured or under heavy load, the spanning port may not see all traffic. This is especially true when multiple ports are spanned to a single port; although possible, this can quickly overload the span port. Moreover, in case of heavy load, spanning might be temporarily disabled. Finally, most switches only have one spanning port, which may not be enough if multiple network monitoring tools are used. A hub may also be used to passively monitor traffic. It is placed between the connections to be monitored, for example a router and a switch. This allows traffic to still flow between them, while the properties of the hub cause a copy of the traffic to be copied off to the IDPS. Despite its simplicity and reduced cost (4-port hubs are very economical), the use of hubs is not recommended. Being a layer-1 device, a hub is inadequate when the connection between the switch and the router is a fullduplex connection, as collisions would degrade the throughput. Similarly, it should not be used when the IDPS sensor communicates with the other IDPS components through the company’s standard networks, since collisions would impact the flow of traffic between the router and the switch. A network tap represents a further solution, indeed very similar to a hub. It creates a direct connection between a sensor and the physical network, usually a fiber optic cable, and provides the 11

Figure 2.2: Network-based IDPS sensors: a passive deployment. sensor with a copy of all network traffic being carried by the media. Unlike hubs, which are prone to failure (especially low-cost ones), taps are by design fault-tolerant, having the main connection hard-wired into the device. They do not impact the traffic flow and, once a tap is in place, changes to the IDPS infrastructure do not influence the overall network. Disadvantages to using taps are that they must be purchased separately and installed, which generally causes some network downtime. Finally, an IDS load balancer is a device that aggregates traffic from one or more spanning ports or network taps and directs it to monitoring devices, such as IDPS sensors. The volume of traffic may be entirely sent to each sensor or it may be split among multiple sensors to balance the load (hence the name) and avoid overwhelming; some load balancers can also be configured to direct traffic to different devices based on IP addresses, protocols and other characteristics, so that specific analysis can be done on certain types of traffic. Splitting traffic among multiple sensors may reduce detection accuracy, because related events or portions of a single attack may be seen by different sensors.

2.1.2

Detection capabilities

Network-based IDPSs typically make use of a combination of the three detection techniques formerly discussed (signature-based, anomaly-based and stateful protocol analysis) in a tightly interwoven way. For example, stateful protocol analysis might be used to decompose traffic into requests-responses pairs and each of them may be examined for anomalies and compared to signatures. They can usually detect reconnaissance activity and potential threats at the application, transport and network layers by analyzing a number of protocols: DHCP, DNS, FTP, HTTP, POP, IMAP, SMTP, IRC, peer-to-peer file sharing protocols, and many others, at the application layer; TCP and UDP at the transport layer; IPv4, ICMP and IGMP at the network layer. Many products are also adding support for IPv6, which can range from very simple forms (such as identifying that IPv6 traffic is present) to full analysis capabilities. Unexpected services, such as tunnelled protocols, backdoors, host running unauthorized application services, are detected through stateful protocol analysis, which can determine if the activity in a connection is consistent with the expected application protocol, or through anomaly detection methods, which can identify changes in network flows and open ports on hosts. Policy violations can be identified by looking at port numbers, IP addresses and website names. 12

Some IDPSs are able to identify which operating system is running on each host through various techniques. For example, the ports used by a host or certain characteristics in packet headers can indicate that a particular OS or an OS family is being used (e.g., Windows, Unix). The same class of techniques, known as passive fingerprinting, can be applied to the application layer to infer which applications (web browsers, web servers etc.), and sometimes which version, are running on a host. Such information may be proficiently exploited by the IDPS to understand whether a given attack is likely to succeed or not, because for example the targeted application has a known vulnerability, and generate high- or low-priority alerts accordingly (attacks are typically stopped in any case, whether or not they could easily succeed).

2.1.3

Prevention capabilities

Depending on the type of sensor deployment, an IDPS solution may offer the following prevention capabilities: • Ending the current TCP session (passive sensors). This is done by sending TCP reset packets to both endpoint, to make it appear to each endpoint that the other endpoint is trying to end the connection. It is sometimes called session sniping. In many cases this technique is ineffective because the reset packets are not received in time. Also, it is only applicable to TCP and therefore cannot stop attacks carried in other protocols. • Performing inline firewalling (inline sensors), which enables the sensor to drop or reject suspicious network activity. • Throttling network usage (inline sensors). It enables to limit the percentage of network bandwidth consumed by a specific protocol, if it is being used inappropriately. This prevents the activity from negatively impacting bandwidth usage for other resources. • Altering malicious content (inline sensors). Some IDPS sensors can identify malicious content in a packet and replace it with benign content. Such sanitization may be performed on headers, to block some attacks involving packet headers or application headers or, additionally, it can strip infected attachments from e-mails, as well as other pieces of malicious content. • Reconfiguring other network security devices (both passive and inline sensors). This prevention technique instructs network security devices such as firewalls, routers and switches to reconfigure themselves to block certain types of activity or route it somewhere else. • Running a third-party program or script (both passive and inline sensors). Some IDPS sensors can run a specified program or script when certain kind of alerts are raised. Third-party programs and scripts are most commonly used when the IDPS does not support the desired prevention action(s).

2.1.4

Detection accuracy and tuning

Historically, network-based IDPSs have been associated with high rates of false positives and false negatives. This is because most of them relied on signature-based techniques; nowadays, the adoption of multiple detection strategies has provided better accuracy. Nonetheless, the number and variety of operating systems, applications and protocols that interact over the monitored network can be large and complex, making it impossible for a sensor to understand everything it sees. Stateful protocol analysis techniques often attempt to replicate the processing of packets and connections that would be done on the endpoint, slightly improving accuracy. Considerable tuning and customization are required to achieve acceptable results. Administrators can specify thresholds for port scans and application authentication attempts, blacklists and whitelists for host IP addresses and usernames. Information regarding the different hosts can be used to refine detection. For example, the IDPS could be instructed on which addresses correspond to a given type of service, such as a web server or a mail server, which version of a given application runs on a host, etc. 13

2.1.5

Limitations

Network-based IDPSs have some significant limitations. Three particularly important ones relate to how the system deals with encrypted traffic, high traffic loads and attacks against the IDPS itself. NIDPSs cannot detect attacks within encrypted network traffic, including Virtual Private Network (VPN) connections, HTTP over SSL (HTTPS) and SSH sessions. Some of them are able to perform an analysis of the initial negotiation conducted when establishing encrypted communications, in order to identify known vulnerabilities or misconfiguration. Encrypted payloads should be analysed before encryption or after decryption, which is possible with traffic that entered an organization through a VPN gateway and has since been decrypted, but impossible in most other cases. A host-based IDPS solution would better suit this need. Under high loads, a network-based IDPS may not be able to perform a full analysis of the traffic and have to drop some packets. For passive IDPS sensors this would result in some incidents going undetected, especially if stateful protocol techniques are in use. For inline sensors, dropping packets would cause disruptions in network availability and delays in processing packets could lead to unacceptably high latency. Many implementations are able to detect high-load conditions and either pass certain types of traffic without performing a full scan or drop low-priority traffic. Vendors typically rate their sensors by maximum bandwidth capacity, but the actual capacity of any product is a function of several parameters, including the protocols in use, the depth of analysis performed for each protocol, the longevity of connections and the number of simultaneous connections. Large volumes of traffic may also indicate distributed denial of service (DDoS) attacks and anomalous activity, such as fragmented packets, intended to exhaust a sensor’s resources or cause it to crash. Other types of attack, known as blinding, could generate traffic that is specially crafted to trigger many alerts in a short period of time, but usually not intended to perform any actual attack. The real attack is run separately at the same as the blinding traffic, in the hope that the IDPS will fail to detect it or that its alert will go unnoticed. Some IDPS sensors, however, are now able to recognize common DDoS and blinding techniques and can ignore the rest of the activity, reducing the load on the sensors.

2.2

Wireless IDPS

A wireless IDPS monitors wireless network traffic and analyzes its wireless networking protocols to identify suspicious activity. The threats that wireless networks and a wired networks have to face are generally the same, but the relative risk of some threats varies significantly. Wireless attacks typically require the attacker to be within close proximity to the wireless network, whereas many attacks on wired networks can be performed remotely from any location. However, many WLANs are configured so that they only require weak forms of authentication, if any, making the job much simpler for local attackers. Man-in-the-middle attacks or attacks that rely on injecting additional messages into network communications can thus be easily carried out.

2.2.1

Components and architectures

The typical components in a wireless IDPS are the same as a network-based IDPS: consoles, database servers, management servers and sensors. Wireless sensors play the same role as NIDPS sensors, but they function very differently because of the complexities of monitoring wireless communications. While NIDPS sensors can see all packets on the network they monitor, wireless sensors work by sampling traffic. There are two frequency bands to monitor, 2.4 GHz (IEEE 802.11b, g) and 5 GHz (IEEE 802.11a), separated into channels. It is not currently possible to monitor all channels in a band simultaneously. A sensor can only monitor one channel at a time; when it is ready to switch to a different channel, it must shut its radio off, change the channel and turn its radio back on. Sensors typically change channels frequently (several times per second), which is known as channel scanning, to avoid overlooking the other channels for long intervals and thus potentially missing out some attacks. Specialized sensors are

14

available to reduce channel scanning; they are equipped with several radios and high-power antennas and each radio/antenna pair is used to monitor a different channel. Wireless sensors are available in multiple forms: dedicated, bundled with an access point and bundled with a wireless switch. Dedicated sensors, which provide better performance than the other types, perform wireless traffic sniffing in a passive way, without passing it from source to destination. Some of them are also capable of analysing the traffic they monitor, while others forward it to a management server for analysis. Two deployments are possible: fixed or mobile. A deployment with dedicated sensors is illustrated in Figure 2.3.

Figure 2.3: A deployment of wireless-based IDPS dedicated sensors. Some vendors have added IDPS capabilities to access points. Such devices divide their time between providing network access and monitoring traffic, which inevitably results in a less rigorous detection capability than a dedicated sensor. If multiple channels or bands need to be scanned, the basic AP functionality will be disrupted, which ultimately makes bundled sensors a less efficient solution. Wireless switches, which are intended to assist administrators with managing and monitoring wireless devices, can also integrate some wireless IDPS capabilities, typically not as strong as bundled APs or dedicated sensors. Wireless IDPS components are typically connected to each other through a wired network; because there should already be a strictly controlled separation between the wireless and wired networks, using either a management network or a standard network should be acceptable. Sensor locations should be chosen so that sensors can monitor the RF range of the organisation’s WLANs. Many organisations also want to deploy sensors to monitor physical regions where no WLAN activity should be present, as well as bands and channels that the organisation’s WLANs should not use, as a way of detecting rogue APs and ad-hoc WLANs.

2.2.2

Detection capabilities

Wireless IDPSs can detect attacks, misconfigurations and policy violations at the WLAN protocol level, primarily examining IEEE 802.11a, b, g and i protocol communication. Higher-level protocols are not taken into account. Some wireless IDPS products only use simple signature-based detection, 15

while others include several detection techniques to achieve broader and more accurate detection. Wireless sensors can detect a number of events, such as unauthorized WLANs and WLAN devices and poorly secured WLAN devices. This is accomplished by identifying deviations from organisationspecific policies for settings such as encryption, authentication, data rates, SSID names and channels. For example, a sensor could detect that WEP is being used instead of WPA 2. This category includes the majority of the events that are ever detected by wireless IDPSs. Some sensors can also use anomaly-based methods to detect unusual WLAN usage patterns, such as irregular amounts of traffic, several failed attempts to join the WLAN or unauthorized WLAN activity during off-hours periods. The use of active wireless network scanners, which aim at identifying unsecured or weakly secured WLANs, can also be detected by some sensors, as well as spoofing the identity of another device. DoS attacks on wireless networks include logical attacks such as flooding, which involve sending large quantities of messages, and physical attacks such as jamming, which emits electromagnetic energy on the WLAN’s frequencies in order to make it unusable. This kind of attacks can be detected through stateful protocol analysis or by counting events such as session terminations over a period of time and alerting when threshold values are exceeded. Once a threat has been detected, most IDPS sensors can identify its location by using triangulation. If IDPS devices can be configured to include information on the organisation’s buildings, sensors might also be able to identify whether a threat is inside or outside the organisation’s facilities.

2.2.3

Prevention capabilities

Wireless IDPS prevention capabilities may be split into two categories: wireless and wired. Wireless prevention actions are performed by means of the sensor itself. Some sensors can terminate connections between a rogue or misconfigured station and an authorized access point or between an authorized station and a rogue or misconfigured access point. This is done by sending messages to the endpoints, telling them to terminate the current association. When a sensor is transmitting signals to terminate connections, it may stop its monitoring activity until the prevention action has been completed; to avoid this, some sensors have two radios, one for detection and one for prevention. Wired prevention, on the other hand, is done by instructing a switch on the wired network to block all activity concerning a given station or access point, based on its MAC address or the switch port it is connected to. This technique only stops the malicious device from communicating with the wired network, but it doesn’t prevent it from pursuing its action through the air.

2.2.4

Detection accuracy and tuning

Compared to other forms of IDPS, wireless IDPS is usually more accurate, due to its characteristics which limit the scope of analysis to wireless networking protocols. For the same reason, not many tunings and customizations are available. False positives can be generated by anomaly-based techniques if threshold values are not properly set. Most wireless IDPSs support blacklists and whitelists of known malicious and benign WLAN devices, which can also be used to record authorized or unauthorized WLAN NIC vendors.

2.2.5

Limitations

While offering rather robust detection capabilities, wireless IDPSs still suffer some limitations. The first type of threat to which wireless IDPSs are exposed is passive monitoring of traffic, i.e. monitoring activity over the wireless network without generating any traffic, which cannot be detected by sensors. The attacker can therefore capture traffic unnoticed, process it offline and, if some weak protection is being used (such as WEP), he can find the encryption key used to provide security for the wireless traffic without much effort. With this key, the attacker can decrypt all traffic from the WLAN. An IDPS cannot fully compensate for the use of insecure wireless networking protocols. Another problem with some wireless IDPS sensors is the use of evasion techniques. Attackers might resort to fingerprinting to infer which IDPS product is in use. Once the product has been identified, evasion techniques can be used to take advantage of the product’s channel scanning scheme. For instance, attacks could be performed in very short bursts on channels that are not currently being 16

monitored or on two different channels at the same time, so that only one can be detected. A further disadvantage of channel scanning, although not related to evasion, is that each sensor only sees a fraction of the activity on each channel, making it considerably more difficult to analyze. Wireless IDPS sensors are also susceptible to attack; the same attacks (logical and physical DoS) that aim at disrupting WLANs can also disrupt sensor functions. They are also easily noticeable and hence vulnerable to physical damage; for that reason, they are often designed to look like fire alarms or regular access points.

2.3

Network Behavior Analysis (NBA) system

A flow is a particular communication session occurring between hosts. A network behaviour analysis (NBA) system examines network traffic or statistics on network traffic to identify unusual flows, such as distributed denial of service attacks, certain forms of malware and policy violations. The difference with a network-based IDPS solution does not seem striking: they serve much the same purposes, with NBA systems making broader use of anomaly-based techniques.

2.3.1

Components and architecture

NBA solutions typically have sensors and consoles; some products also include management servers, which are sometimes called analyzers. Sensors usually come in the form of appliances. Some of them are similar to NIDPS sensors and sniff traffic directly from one or more network segments; others do not monitor the networks directly, but instead rely on network flow information provided by routers and other networking devices. Many standards exist for flow data formats, such as NetFlow and sFlow. Typical flow data particularly relevant to intrusion detection and prevention includes source and destination IP addresses, source and destination TCP and UDP ports, ICMP types and codes, number of packets and number of bytes transmitted in the session, timestamps for the start and end of the session. A separate management network may be used to interconnect the various components; if sensors collect flow information from other devices, rather than from the networks themselves, the NBA solution can be logically separated from the rest of the organisation’s networks. An example of an NBA network architecture is given in Figure 2.4. Sensors are usually deployed in passive mode, using one of the connection methods outlined for network-based IDPS (network tap, spanning port), and should be placed to monitor key network locations, such as the divisions between networks, or key network segments, such as DMZ subnets.

2.3.2

Detection and prevention capabilities

NBA technologies can detect several types of malicious activity. Most of them use primarily anomalybased detection, along with some stateful protocol analysis techniques; signature-based detection is usually not available. The types of threat that can be identified include: denial of service attacks, by noticing significantly increased bandwidth usage or unusual data flows to/from specific hosts; scanning, which generates atypical flow patterns; worms, which may generate large amounts of traffic, cause hosts to communicate with each other or use ports that are normally not used; unexpected application services, detected through stateful protocol analysis methods; policy violations. Most NBA systems can also reconstruct a series of events to determine the origin of a threat, such as identify the host on the organisation’s network that first transmitted a worm to the others. The prevention capabilities offered by NBA solutions are not substantially different from those provided by network-based IDPSs. Since sensors are typically deployed passively, prevention actions range from session sniping to reconfiguring other network devices or running third-party scripts or programs.

17

Figure 2.4: An NBA IDPS deployment.

2.3.3

Detection accuracy and tuning

NBA solutions are strongest at detecting significant deviations from normal behaviour, such as unusually large amounts of traffic or anomalous flow patterns. They are less accurate at identifying small-scale attacks, especially if they are conducted slowly. NBA sensors can take some time to label an activity detected on the networks as abnormal behaviour, so many attacks do not trigger any alarm until they reach a point where their activity is significantly different from what is expected. As a consequence, NBA systems may not be the ideal solution when a timely response to attacks is fundamental. Sensors can usually be tuned and made more sensitive to anomalous activity, which means alerts will be generated more promptly when attacks occur; however, this will increase the number of false positives. The latter can also be caused by benign changes in the environment, such as adding a new service or moving an existing service to a different host. Besides sensor sensitivity, not much tuning is usually required. NBA products automatically update their inventories of expected flows and host characteristics on an ongoing basis. Administrators might periodically adjust thresholds, update firewall ruleset-like policies and whitelists and blacklists of hosts and services.

2.3.4

Limitations

As mentioned in the previous section, the main limitation to NBA solutions is the delay in detecting attacks. Some delays are inherent in anomaly detection methods which, by their own nature, need time to assess purported deviations from normal behaviour. However, NBA technologies often suffer from additional delays caused by their data sources, especially when sensors rely on flow data from routers and other networking devices. As a matter of fact, this data is often transferred to the NBA system in batches; depending on the product and on its configuration, this could occur relatively frequently (e.g., every minute) or relatively infrequently (e.g., every 15 minutes). Chances are, when long delays are present, that an attack be detected once it has already disrupted or damaged systems.

18

2.4

Host-based IDPS

Unlike other types of IDPS solutions, which are intended to monitor all traffic on a network, regardless of its source, a host-based IDPS monitors the characteristics of a single host and the events occurring within that host for suspicious activity. A host-based IDPS could for example monitor incoming and outgoing wired and wireless network traffic, system logs, running processes, file access and modification, system and application configuration changes. The same kind of monitoring is often provided by a number of anti-virus software and personal firewall solutions. The distinction between these tools becomes very blurred, as many of them overlap in functionality.

2.4.1

Components and architecture

Host-based IDPSs include agents, which are the equivalents of sensors and transmit data to management servers, which in turn may optionally use database servers for storage. Consoles are also used for management and monitoring. In some cases, agents are software-only; this means they are installed on the host they guard as any other application and interact with the operating system. Alternatively, some products use dedicated appliances running agent software instead of installing agent software on individual hosts. In that case, each appliance is positioned to monitor traffic going to and from a particular host. These appliances are very similar to network-based IDPS devices, but they usually offer monitoring capabilities for only one specific type of application, such as a web server or a database server, so they are more specialized than standard network-based IDPSs. Advantages and disadvantages to each type of agent will be discussed later. Agents are typically designed to protect server or client hosts or, sometimes, they can perform monitoring for a specific application service only. This type of agent is also known as an applicationbased IDPS. Other types of network devices, such as firewalls, routers and switches, cannot normally include a host-based agent. The network architecture for host-based IDPS deployment is rather simple. The components usually communicate by means of the organization’s existing networks instead of using a separate management network. This is to avoid having to equip the guarded hosts with multiple network interfaces and to ensure that traffic cannot pass from one interface to the other. Most products encrypt their communications. Appliance-based agents are typically deployed inline immediately in front of the hosts that they are protecting. Figure 2.5 shows an example of a host-based IDPS deployment architecture. Host-based IDPS agents are most commonly deployed to critical hosts, such as publicly accessible servers and servers that contain sensitive information. Agents being available for a number of server and desktop/laptop operating systems, some organizations could decide to install them on most of their hosts. In addition, host-based IDPS agents are often used because they can analyze activity that cannot be monitored by other security controls; an example of that is activity within encrypted network communications, which cannot be analyzed by common network-based sensors. Host-based agents, on the other hand, are installed on endpoints and can analyse traffic before encryption and after decryption.

2.4.2

Host agents architecture

Most software agents alter the internal architecture of the hosts they monitor in order to provide intrusion prevention capabilities. This is typically done through a shim, which is a layer of code interposed between existing layers of code. A shim intercepts data at a point where it would normally be passed from one piece of code to another; it can analyze it and determine if the action should be permitted or not. Shims are used to control several types of resources, such as network traffic, filesystem activity, system calls, Windows registry activity and common applications. The use of shims enables the IDPS to have a rather accurate overview of the activity occurring on the host. Nevertheless, intercepting and analysing all activity on a host can be very resource intensive

19

Figure 2.5: A host-based IDPS deployment. and add a significant strain on the system, negatively impacting its performance. For that reason, some host-based IDPS do not alter the host architecture; instead, they monitor activity without shims, by analysing application log entries and file modifications. Although less intrusive and more lightweight to the host, these methods are also generally less effective at detecting threats and often cannot perform many prevention actions. The principle of operation of a host-based agent depends on the fact that successful intruders will generally leave a trace of their activities. Such intruders will often establish their ownership by installing software that will grant them future access to carry out whatever activity the envisage, such as keystroke logging, identity theft, spamming, botnet activity, spyware-usage. These traces could be identified by a skilled computer user; the IDPS automates such a task. In general, the agent will create a database of system objects it should monitor, usually file-system objects, and check that particular regions of memory have not been modified, like the system-call table. For each object at stake the agent will remember its attributes (permissions, sizes, modification dates) and create a checksum for the contents (MD5, SHA-1 hash or similar). This information will then be stored in a database and secured. Since the process of initialising the checksum database generally requires a long time and involves cryptographically locking each monitored object, manufacturers usually design the object database in such a way that frequent updates are not necessary. Computer systems also have many dynamic, frequently changing objects, whose nature makes them unsuitable for checksum techniques; a raft of other means are then used to monitor changes. Once the object and checksum databases have been properly constructed, the agent regularly scans the monitored objects, reports on anything that may appear suspicious and stops potentially dangerous actions. Since intrusions might still succeed, the IDPS databases are just as likely to be tampered as the objects they guard. For that reason, some IDPSs might allow administrators to store the databases on a read-only memory device, such as a cd-rom. Similarly, logs produced by the IDPS are often sent off-system immediately, sometimes via one-way communication channels.

20

2.4.3

Detection and prevention capabilities

Most host-based IDPSs use a combination of signature- and anomaly-based detection techniques. Specific techniques that are commonly used include: • Code analysis. These techniques aim at identifying malicious activity by analysing attempts to execute code. For example, code behaviour analysis can first execute code in a virtual environment and compare its behaviour to profiles or rules; buffer overflow detection identifies typical sequences of instructions that attempt to perform stack and heap buffer overflows; other techniques are able to monitor system calls and restrict which drivers and dynamic link libraries (DLL) can be loaded. • Network traffic analysis and filtering. This is often similar to what a network-based IDPS does. It analyses network, transport and application layer protocols and includes processing for common applications. Agents often include a host-based firewall that can restrict incoming and outgoing traffic for the system. • Filesystem monitoring. It includes a number of methods, such as file integrity checking, which generates checksums for critical files and periodically verifies them, and file attribute checking; these two methods can only determine after-the-fact that a file has been changed. Some agents, typically those which use a shim, are able to monitor all attempts to access critical files and stop attempts that are suspicious. The current attempt is compared against a set of policies regarding file access and blocked if the type of access that has been requested (read, write, execute) contradicts a policy. • Log analysis. Some agents can identify malicious activity by monitoring and analyzing system and application logs, which contain information on events such as shutting down the system, starting a service, as well as application startup and shutdown, failures, major configuration changes. • Network configuration monitoring. Agents are able to monitor a host’s current network configuration and detect changes to it. For example, network interfaces being placed in promiscuous mode, additional TCP or UDP ports or unusual protocols being used (such as non-IP protocols) could indicate that the host has already been compromised and is being configured for use in future attacks or for transferring data. • Removable media restriction. Some products can enforce restrictions on the use of removable media, which can prevent malicious content from being transferred to a host and stop sensitive files from being copied from the host. • Process status monitoring. Some host-based IDPSs can monitor the status of the processes and services running on a host; when they detect that one has stopped, they restart it automatically. This provides protection against some forms of malware which can sometimes disable antivirus software and the like. • Network traffic sanitization. This form of protection is usually implemented by appliancebased IDPSs. Sanitization of traffic may rebuild all requests and responses directed to the host or coming from it, thus neutralizing certain unusual activity, particularly in packet headers and application protocol headers. It can also reduce the amount of reconnaissance the attackers can perform on the host, by hiding OS fingerprints and application error messages.

2.4.4

Detection accuracy and tuning

The accuracy of detection for host-based IDPS solutions is generally more challenging than for the other types of IDPSs. While the events occurring on the guarded host are detected rather accurately, their benign or malicious nature is often hard to determine. This is due to the fact that agents sometimes lack sufficient context information. Some of them, particularly those intended for desktop 21

or laptop use, prompt users to provide context, such as whether or not the user intentionally started an update or made a configuration change. Considerable tuning may be required to improve detection accuracy. Some agents which make use of anomaly-based techniques might need to observe host activity during a certain period of time in order to develop profiles of expected behaviour. Others need to be configured with detailed policies which define, on a per-application basis, which operations should be permitted and what should be considered as normal behaviour. Other customization capabilities include the definition of blacklists and whitelists for hosts, applications, ports, filenames and other objects.

2.4.5

Limitations

Host-based IDPS have significant accuracy limitations, as has already been considered. Another drawback is represented by the fact that, unlike the other IDPS technologies, with host-based IDPSs the guarded system and the guardian coincide. This means that agents consume the host’s resources, such as memory, processor usage and disk storage, especially if they make use of shims. When this would slow down performance in an unacceptable way, IDPS functionalities can be delegated to an appliance-based agent. This solution, despite leaving the host’s resources free, usually provides less accurate and in-depth detection. Delays in generating alerts are another weakness of host-based IDPSs. This may be due to some detection techniques which only scan the system for threats a few times a day. What is more, many agents are intended to forward their alert data to the management servers on a periodic basis and not in real-time. For example, alert data may be transferred in batches every 15 to 60 minutes to reduce overhead for the IDPS components and the network. This can cause delays in initiating response actions, which increases the impact of incidents that spread quickly, such as malware infestations.

2.5

Honey pots

As outlined in section 1.1, one of the purposes of IDPSs is to gather information relative to the types of threats an organisation receives and to study attackers’ techniques and aims. Once sufficient information has been acquired, it can be used to harden security on the company’s servers. A way of collecting such information without compromising the assets of the company is using honey pots. As their name suggests, honey pots are systems intended to lure hackers by exposing known vulnerabilities deliberately. They are valuable as a surveillance and early-warning tools, but they should have no production value and hence should not get in touch with any legitimate traffic, activity or data. It is important, however, that honey pots look like real systems and appear to be part of a network. Fake data files, user accounts and the like should be created to reassure a hacker. There are different ways to build and place honey pots. A honey pot should have common services running on it, such as a Telnet server, a web server, an FTP server and so on. It could be placed somewhat near real production servers, so that it can be confused with a real server, for example by assigning contiguous IP addresses to them. Firewalls or routers could also be configured to redirect traffic on some ports to a honey pot. Although useful, a honey pot may carry risks to a network and must be handled with care. If not properly isolated, a honey pot could be used by an attacker to break into real systems. An organisation should deploy a honey pot only if it has the resources both to manage it and to inspect the collected data and extract valuable information from it.

22

Chapter 3

Snort: an open-source NIDPS
Snort is a free and open-source network-based intrusion detection and prevention system. It is a software-based NIDPS. Originally written in 1998 by Martin Roesch, today it has arguably become a “de facto standard in (network) intrusion detection and prevention” – exactly what the product prides itself on being. It is now developed and supported by Sourcefire, of which Roesch is the founder and CTO. Snort performs protocol analysis and content searching and matching; it is commonly used both to actively block and passively detect a variety of attacks and probes.

3.1

Modes of operation

Snort can be configured to run in three modes: sniffer/packet logger mode, network intrusion detection system mode and inline mode. When in the sniffer mode, Snort simply reads the packets from the network and displays them in a continuous stream on the console. The packet logger mode does the same thing, except for logging the packets to disk. The network intrusion detection system mode is the most complex and widely configurable mode. It allows Snort to analyse network traffic for matches against a user-defined rule set and performs several actions. The inline mode is designed to obtain packets from iptables, rather than from libpcap (a library that provides a system-independent API for low-level network monitoring). It can then cause iptables to drop or pass packets based on Snort rules.

3.2

Components

Snort is logically divided into multiple components, which work together to detect particular attacks and to generate output in a given format. The major components are the packet decoder, the preprocessors, the detection engine, the logging and alerting system and the output modules. A brief overview of each component is given in the paragraphs that follow. Figure 3.1 illustrates the architecture of Snort. Packet decoder. The packet decoder’s role is simple: it takes packets from different types of network interfaces and prepares them to be pro-precessed or to be sent to the detection engine. The interfaces may be Ethernet, SLIP, PPP and others. Preprocessors. Preprocessors are components or plug-ins that can be used with Snort to arrange or modify data packets before they are analysed by the detection engine. Some preprocessors also perform some form of detection by identifying anomalies in packet headers and generating alarms. Preprocessors are very important when packets are compared against a set of rules, as it is the case with Snort. Hackers may use a number of evasion techniques to bypass detection, as explained in section 1.6. Some of these techniques can be effectively countered by preprocessors. They can, for 23

Figure 3.1: Architecture of the Snort NIDPS. instance, analyse and decode strings containing hexadecimal or Unicode characters to avoid character encoding-based evasion, parse strings that include paths, defragment packets, reassemble TCP streams etc. Detection engine. The detection engine is the core of the Snort IDPS. Its responsibility is to detect if any intrusion activity exists in a packet. Snort employs sets of rules to do so; rules are written in files by the administrator and copied to internal data structures, or chains, at load time. If a packet matches any rule, appropriate action is taken, otherwise the packet is dropped. Appropriate actions may be logging the packet or generating alerts. The detection engine is the time-critical part of Snort. Depending on the power of the machine that is running it, on how many rules have been defined and on the load of the network, it may take different amounts of time to respond to different packets; when network traffic is too high this could result in dropping some packets and not getting a true real-time response. Once the choice of the machine on which to deploy Snort has been made, efforts should be made to catch as many signatures as possible with the smallest number of rules. The detection engine can dissect a packet and apply rules on different parts, such as the different headers (IP, transport, application) or the packet payload. In all 1.x versions, the detection engine would stop further processing of a packet once a rule was matched and action would be taken according to the rule. This meant that if a packet matched criteria defined in multiple rules, only the first rule to be matched in the chain was applied, regardless of the other rules. This behaviour represented a security problem: if a low-priority priority rule was matched first, all subsequent rules were ignored, even if high-priority. As of version 2, the problem has been rectified by matching all possible rules against a packet and, only then, generating an alert, based on the highest-priority matched rule. Logging and alerting system. Depending upon what the detection engine finds inside a packet, the latter may be used to log the activity or generate an alert. Logs may be produced in the form of simple text files, in tcpdump-style files or some other form. The tcpdump binary mode usually allows higher logging speed and can be understood, for later analysis, by common sniffer programs, such as tcpdump or Ethereal. Command line options are available to modify the type and detail of information that is logged. Alerts can be generated in one of many ways; for example, they can be included in the log files, sent to a UNIX socket where another program is listening or displayed in a user console. Output modules. Output modules or plug-ins process alerts and logs and can perform different operations. Depending on the desired type of output or response, output modules can do things as simple as logging to the default folder, sending messages to the syslog facility, or more complex such as logging to a database, like MySQL or Oracle, and modifiying the configuration on routers and firewalls. 24

Conclusions
The four primary types of IDPS technologies described in this report each offer fundamentally different information gathering, logging, detection, and prevention capabilities. Each technology type offers benefits over the other, such as detecting some events that the others cannot, detecting some events with significantly greater accuracy than the other technologies, and performing in-depth analysis without significantly impacting the performance of the protected hosts. Organizations should consider using multiple types of IDPS technologies to achieve more comprehensive and accurate detection and prevention of malicious activity, with lower rates of false positives and false negatives. Intrusion detection systems are evolving products. Research began in the mid-1980s and, since the appearance of the first products in the mid-1990s, this area has continued to change as new research influences the design of products. Recent trends have shown to be headed towards the integration of intrusion detection technology within the TCP/IP stack, yielding the so called stack-based IDPSs.

25

Bibliography
[1] K. Scarfone, P. Mell – Guide to Intrusion Detection and Prevention Systems (IDPS). NIST SP 800-94, 2007. http://csrc.nist.gov/publications/nistpubs/800-94/SP800-94.pdf [2] C.P. Pfleeger, S.L. Pfleeger – Security in computing. Fourth Edition, 2006. [3] W. Stallings – Cryptography and Network Security Principles and Practices. Fourth Edition, 2005. [4] A. Brox – Signature-Based or Anomaly-Based Intrusion Detection: The Practice and Pitfalls. http://www.scmagazine.com/asia/news/article/419779/ signature-based-anomaly-based-intrusion-detection-practice-pitfalls [5] E. Hacker – IDS Evasion with Unicode. http://www.securityfocus.com/infocus/1232 [6] Intrusion Detection System (IDS) Evasion. An iDefense Security Report, 2006. http://complianceandprivacy.com/WhitePapers/iDefense-IDS-Evasion/ iDefense IDSEvasion 20060510.pdf [7] Snort Team – Snort User’s Manual 2.8.0. http://www.snort.org/docs/snort manual/2.8.0/snort manual.pdf [8] B. Laing – How To Guide: Implementing a Network-Based Intrusion Detection System http://www.snort.org/docs/iss-placement.pdf [9] C. Viklund – Identifying Threats in a Wireless Environment. Master’s thesis, Lule˚ University of a Technology, Sweden. http://epubl.ltu.se/1402-1617/2005/234/LTU-EX-05234-SE.pdf [10] R.U. Rehman – Intrusion Detection Systems with Snort. http://www.informit.com/content/downloads/perens/0131407333.pdf

26

Sign up to vote on this title
UsefulNot useful