Intrusion Detection and Prevention Systems

Alberto Grand
Contents

Introduction 2

1 Overview 3
1.1 Purposes of IDPSs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3
1.2 Accuracy of IDPSs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3
1.3 Classification . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4
1.4 Common detection methodologies . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4
1.4.1 Signature-based detection . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4
1.4.2 Anomaly-based detection . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5
1.4.3 Stateful-protocol analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6
1.5 IDPS architectures and components . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6
1.6 Evasion techniques . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7

2 Types of IDPS technologies 10

2.1 Network-based IDPS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10
2.1.1 Components and architectures . . . . . . . . . . . . . . . . . . . . . . . . . . . 10
2.1.2 Detection capabilities . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12
2.1.3 Prevention capabilities . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13
2.1.4 Detection accuracy and tuning . . . . . . . . . . . . . . . . . . . . . . . . . . . 13
2.1.5 Limitations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14
2.2 Wireless IDPS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14
2.2.1 Components and architectures . . . . . . . . . . . . . . . . . . . . . . . . . . . 14
2.2.2 Detection capabilities . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15
2.2.3 Prevention capabilities . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16
2.2.4 Detection accuracy and tuning . . . . . . . . . . . . . . . . . . . . . . . . . . . 16
2.2.5 Limitations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16
2.3 Network Behavior Analysis (NBA) system . . . . . . . . . . . . . . . . . . . . . . . . . 17
2.3.1 Components and architecture . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17
2.3.2 Detection and prevention capabilities . . . . . . . . . . . . . . . . . . . . . . . . 17
2.3.3 Detection accuracy and tuning . . . . . . . . . . . . . . . . . . . . . . . . . . . 18
2.3.4 Limitations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18
2.4 Host-based IDPS . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19
2.4.1 Components and architecture . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19
2.4.2 Host agents architecture . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19
2.4.3 Detection and prevention capabilities . . . . . . . . . . . . . . . . . . . . . . . . 21
2.4.4 Detection accuracy and tuning . . . . . . . . . . . . . . . . . . . . . . . . . . . 21
2.4.5 Limitations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22
2.5 Honey pots . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22

3 Snort: an open-source NIDPS 23

3.1 Modes of operation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23
3.2 Components . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23

Conclusions 25

1
Introduction

Intrusion is one of the most widely known threats to security in computer and network systems and
has existed as long as the systems themselves. Back in the 1980s, James Anderson published a
paper in which he pointed out that audit trails contained vital information that could be valuable in
tracking misuse and understanding user behaviour, thereby laying the foundations of modern intrusion
detection concepts. He defined an intrusion as a possibly deliberate unauthorized attempt to access
information, manipulate information or render a system unreliable or unusable. An ideal, perfectly
secure system should not allow intrusion to occur; in practice, no real system being completely secure,
intrusion attempts can succeed. It is therefore important to detect them and limit as much as possible
the damage they can produce. Several intrusion detection techniques have been devised for that
purpose.
The word intrusion detection refers to the process of monitoring all activity occurring in a network
or a computer system searching for signs of possible incidents, which are violations, or threats of
violation, of computer security policies, acceptable use policies or standard security practices. Such
incidents may have different origins. They may be intentionally produced by malicious users outside a
local network who are trying to gain unauthorized access to a system through the Internet, to prevent
a system from providing a given service or to cause unnoticed information leakage from a system.
They might also originate from authorized users of a system misusing their privileges or trying to gain
additional privileges. However, studies have shown that a large amount of computer security incidents
are caused by insiders that are inadvertently violating a security policy. Benign intruders, such as
a user exploring an unauthorized website, might not be particularly harmful and could be tolerable,
although they do consume resources and slow performance for legitimate users. Nonetheless, there is
no way to know in advance whether an intruder will be benign or malign; as a consequence, there is
a motivation for controlling this problem.
Some of the attacks coming from the first category of users may still be countered by using a
firewall, yet not all: some attacks may not violate any specific firewall policy or rule and therefore be
allowed. What is more, a firewall becomes completely useless when attacks, whether intentional or
not, are coming from insiders; hence the need for monitoring and analyzing the events occurring in a
given system and deploying real-time solutions when an incident is detected. An Intrusion Detection
System (IDS) is a piece of software or an appliance that automates the intrusion detection process.
An Intrusion Prevention System (IPS) offers all the functionalities of an IDS, but it additionally
attempts to stop possible incidents. IPS technology is considered by some to be an extension of IDS
technology; NIST’s special publication [1] concerning IDS and IPS technology resorts to the acronym
IDPS (Intrusion Detection and Prevention System) which, although not in widespread use in the
security community, well accounts for the tight connection between these two technologies.

2
Chapter 1

Overview

1.1 Purposes of IDPSs

The primary purpose of an IDPS is the detection of possible incidents. For example, an IDPS could
produce logs of the activity on a given network, identify when an attack is being deployed, implement
the appropriate countermeasures and report the incident to security administrators. Many IDPS can
also be configured to recognize violations of an organisation’s security and acceptable use policies,
for example transfers of inappropriate material throughout the company’s network or downloads of
large databases onto laptops. Some IDPS are also able to identify reconnaissance activity, which may
indicate that an attack is imminent. For example, some forms of malware may perform port scans in
order to identify possible targets for an attack. An IDPS might be able to block reconnaissance activity
and notify security administrators, who may then enable other security controls and counteract the
attack.
In addition to identifying security incidents and supporting incident response efforts, IDPS tech-
nology may serve a number of other purposes:

• Identifying security policy problems. An IDPS could be configured with a set of firewall-
like rules and perform checks on the traffic, notifying the administrators when traffic that should
have been blocked by the firewall but was not, probably due to misconfiguration, is detected.

• Documenting the existing threats to an organisation. Intrusion detection enables the

collection of information that proves useful in understanding intrusion techniques and the fre-
quency and characteristics of the threats to which an organisation is exposed and in tailoring
the security measures accordingly.

• Deterring individuals from violating security policies. Being aware that their activity
is being monitored, individuals might be less prone to violate the organisation’s policies out of
fear of being detected.

1.2 Accuracy of IDPSs

A feature common to all IDPS technologies is that they cannot assure completely accurate detection
of threats. Two types of mistakes can occur: false positives and false negatives. The expression false
positive, which is in use in a variety of other fields, besides computer security, indicates a situation in
which a test (e.g. intrusion detection, anti-virus scan, HIV test) turns out to be erroneously positive.
Conversely, a false negative occurs when a test fails to detect suspicious activity (or a virus, a disease
etc.).
Figure 1.1 illustrates, in very abstract terms, the typical behaviour of an authorized user (right
curve) and of an intruder (left curve). As suggested by the partial overlap between the two curves, it
is not possible to eliminate all false positives and negatives; in most cases, reducing the occurrences
of one increases the occurrences of the other. Sometimes organizations choose to reduce the number
of false positives, so that more potentially malicious events are detected. This way, deeper analysis

3
is necessary to discern true malicious events from false positives. This may seem harmless enough;
however, high rates of false positives can damage productivity due to temporary system down-times
and require a lot of resources. Altering the configuration of an IDPS to improve its accuracy is known
as tuning.

Figure 1.1: Authorized user behaviour and intruder behaviour profiles.

1.3 Classification
IDPS technologies may be classified according to different parameters, namely:

• the methodology(ies) they employ to detect intrusions: signature-based detection, anomaly-

based detection and stateful protocol analysis. Common detection methodologies will be pre-
sented in section 1.4.

• the functionalities they provide, which ultimately differentiate passive systems (IDSs) from re-
active systems (IPSs).

• the type of events they monitor, which are closely related to the type of systems they guard: a
wired network, a wireless network or a single host. In addition to these, a fourth type of IDPS
may be identified, which is known as Network Behaviour Analysis (NBA) IDPS. Types of IDPSs
will be discussed in chapter 2.

1.4 Common detection methodologies

The primary classes of detection methodologies are signature-based detection, anomaly-based detec-
tion and stateful protocol analysis. Many IDPS technologies integrate multiple techniques to provide
more accurate detection. An overview of the three methodologies, along with their major drawbacks
and limitations, is given in sections 1.4.1 through 1.4.3.

1.4.1 Signature-based detection

A signature is a pattern which characterises a known threat. Examples of signatures include the
following: a Telnet attempt with a username of root, which is a violation of an organization’s security
policy; a series of TCP SYN packets sent to many different ports in rapid succession, which may

4
indicate that a port scan is being performed; an e-mail with a subject of “Free ringtones” and an
attachment filename of “ringtones.exe”, which are characteristics of known forms of malware.
Signature-based detection lies in comparing observed events against a set of known signatures to
identify possible incidents. Much like an anti-virus scanner, detection capabilities depend on receiving
regular signature updates, to keep in touch with variations in hacker technique and new types of
threats as they are developed. While being very good at detecting known threats, this technique is
largely ineffective at detecting previously unknown threats. What is more, signature-based solutions
are relatively easy to deceive by slightly varying the attack pattern. For instance, if an attacker
changed the attachment filename in the previous example to “ringtones2.exe”, a signature looking for
“ringtones.exe” would not match it.
There are other problems that affect the accuracy of signature-based detection. Firstly, a compre-
hensive signature database, while providing protection against a broader range of threats, inevitably
increases the computational load for the system charged with analyzing each signature. When the
amount of traffic to be searched becomes substantial, some packets may have to be dropped, thus
potentially overlooking some threats. To avoid this, traffic feeds may have to be split, analysed by
different systems and recombined after analysis, increasing complexity and cost.
Secondly, signature-based techniques only compare the current unit of activity, such as a packet
or a log entry, to a list of signatures and therefore have little understanding of many network and
application protocols. They cannot track the state of complex communications or pair a request
with the corresponding response. This limitation prevents signature-based detection methods from
identifying attacks that comprise multiple events if none of them individually contains a clear indication
of an attack.
Finally, threats disguised by means of evasion techniques are usually not detected by signature-
based methodologies. Evasion aims at modifying the attack in such a way that its appearance is
different, but its effects are the same. For example, an attacker could encode text characters in a
particular way, knowing that the destination host understands the encoding, whereas the IDPS might
not. Section 1.6 includes examples of common evasion techniques.

1.4.2 Anomaly-based detection

The foundations of this detection technique lie in comparing the events observed on a network or a
host with what is considered to be normal activity for them, in order to identify significant deviations.
Two approaches are possible: threshold detection and profile detection.
The first, more simplistic one involves defining thresholds, independent of users, for the frequency
of occurrence of various events. Occurrences of specific events are then counted over an interval of time;
if the count surpasses what is considered a reasonable number, then intrusion is assumed. Threshold
analysis by itself is a largely inaccurate technique; thresholds and time intervals must be fixed, but
because of the variability across users, hosts etc. Such thresholds are likely to generate a lot of false
positives or a lot of false negatives.
Profile-based detection entails the use of profiles that represent the normal behaviour of such
entities as host, users, networks or applications. Statistical methods are then used to identify significant
deviations from profile-recorded behaviour. Profiles are developed by monitoring the characteristics of
typical activity and take into account a number of attributes (number of e-mails sent by a user, number
of failed login attempts for a host, level of processor usage, percentage of available bandwidth used for
web activity during workday hours, etc.). An initial profile is generated over a period of time (typically
days, sometimes weeks) sometimes called a training period. Profiles can be either static or dynamic.
Once generated, a static profile is unchanged unless the IDPS is told otherwise; because typical activity
can change, static profiles eventually become obsolete and therefore need to be regenerated periodically.
On the other hand, dynamic profiles are adjusted on a regular basis as additional events are observed,
so they cannot become outdated. However, their adaptation capabilities make them susceptible to
evasion attempts from attackers. For example, an attacker could perform small amounts of malicious
activity once in a while and gradually increase the frequency and quantity of such activity over time.
If the change is sufficiently slow, the IDPS might regard malicious activity as normal behaviour and
include it in its profile. Inadvertently including malicious activity as part of a profile might also occur

5
during the training period, when the initial profile is built. In some cases administrators can manually
remove parts of a profile that are known to represent malicious activity.
Another problem with building accurate profiles is that, computing activity being very complex, it
is virtually impossible for an IDPS to observe all instances of legitimate behaviour. As a result, some
events which deviate from the recorded profile may trigger alerts and thus generate false positives.
Furthermore, due to the complexity and large number of logged events, it could be difficult for analysts
to understand why a particular alert was generated and validate that it is accurate and not a false
positive.

1.4.3 Stateful-protocol analysis

Stateful-protocol analysis is to some extent similar to anomaly-based detection. It differs from the
latter in that it does not use network- or host-specific profiles generated by the IDPS itself; instead,
stateful-protocol analysis relies on vendor-developed universal profiles that specify how particular pro-
tocols should and should not be used. Profiles enable the IDPS to understand and track the state
of network, transport and application protocols that have a notion of state. It can thus identify
unexpected sequences of commands, such as issuing the same command repeatedly or issuing a com-
mand without first issuing another command upon which it is dependent. For protocols that require
authentication, the IDPS can keep track of the authenticator used for each session and record an au-
thenticator used for suspicious activity, which is helpful when investigating an accident. Some IDPSs
are also capable of distinguishing among multiple users or classes of users to define acceptable activity
differently.
Stateful-protocol analysis could for example keep track of a File Transfer Protocol (FTP) session.
When the session is started, it is initially in the unauthenticated state. Unauthenticated users should
only perform a limited range of commands; unexpected commands might be a symptom of malicious
activity. An important part to understanding state is pairing requests with responses; so, for example,
when an FTP authentication attempt occurs, the IDPS can determine if it was successful from the
status code found in the corresponding response.
Protocol analysis includes reasonableness criteria for individual commands, such as minimum and
maximum lengths for arguments and the presence of binary data within argument strings.
Stateful-protocol analysis methods use protocol models which are typically based on protocol
standards from software vendors and standard bodies, such as IETF or RFC. They usually also
take into account variances among implementations. However, many standards are not exhaustively
complete in explaining some minor details of the protocol; in addition, many vendors either violate
standards or add proprietary features which override standard features and complete details are often
unavailable. As a result, an IDPS may find it extremely difficult to perform accurate, comprehensive
analysis or its interpretation of a protocol may conflict with the way it is implemented in some
applications or operating systems on the end host.
Other drawbacks to stateful-protocol analysis are that it is very resource-intensive and it cannot
detect attacks that do not violate the characteristics of generally acceptable protocol behaviour.

1.5 IDPS architectures and components

All four types of IDPS solutions share the same basic components, although not all of them are
fundamental in all cases. Major components include the following:

• Sensors or agents. They are in charge of monitoring and analyzing activity. The word sensor
normally applies to IDPSs that monitor networks, while agent is typically used for host-based
IDPSs. In the former case sensors could be software programmes, but they are usually ad-hoc
appliances, whereas in the latter case an agent is a piece of software running on the host it
guards.

• Management servers. They are centralized devices that manage the sensors or agents, receive
information collected by the sensors and process it. Some types of sensors or agents are able

6
 to perform basic analysis of the events they detect; management servers, on the other hand,
can perform several kinds of analysis and correlate events by matching information coming
from different sensors. Management servers are available both as appliances and software-only
products. Small IDPSs solutions may not need a management server, but most do include one;
larger-scale deployments usually have multiple management servers and in some cases there are
two tiers of them.
• Database server. It is used as a repository for event information recorded by the sensors or
agents or processed by the management servers. It is not a vital part of an IDPS, although many
IDPS solutions support it.
• Console(s). A console is a software programme that provides an interface for the IDPS’s users
and administrators. It is typically installed on a desktop or laptop computer and can provide
administration-only functionalities, such as configuring sensors or agents and applying software
updates, or both administration and monitoring capabilities. User consoles typically provide
monitoring-only capabilities.
IDPS components may be interconnected in a number of ways, which provide different trade-offs
between level of security and complexity of implementation.
The simplest way to interconnect IDPS components is through the organisation’s standard internal
networks. No additional wires or devices (besides the IDPS itself) need to be set up. This solution is
easy and inexpensive, but it is the least secure of all for a number of reasons. Firstly, IDPS components
need to communicate and they do it using the same network they are guarding; as a consequence,
they must have an address on that network, which reveals their presence to the rest of the network
and makes them possible targets of an attack. Secondly, communication between IDPS components
should be relatively fast and reliable, but there is no such assurance if internal traffic is intense or
under adverse conditions; for example, if the organisation is being subject to a DDoS or worm attack,
there might not be sufficient bandwidth for the IDPS to function properly, which ultimately makes it
useless.
A better solution is the use of a virtual local area network (VLAN) for IDPS management. The
virtual management network still relies on the organisation’s standard network, but it is logically sep-
arated from the latter and virtually invisible to other external users. However, misconfiguration of the
VLAN may lead to the exposure of IDPS data. Furthermore, as with the first solution, incidents that
saturate the organisation’s primary networks still negatively impact the availability and performance
of the IDPS.
A separate management network, strictly designed for security equipment management, provides
an optimal solution. When a management network is used, each sensor has an additional network
interface, known as a management interface, that connects to the management network; the sensor
is unable to pass any traffic between the two interfaces. This configuration is also known as stealth
mode. Other components, such as the management server and the database server, are attached to
the management network only. This architecture effectively isolates the management network from
the production network. IDPS components can operate in a transparent way, without the need for an
IP address on the primary network and can thus conceal their existence from attackers. In case of an
attack, adequate bandwidth is available for the IDPS components to function appropriately. Disad-
vantages to this solution include the additional costs in networking equipment, since a special network
is devoted to IDPS interconnections, and the inconvenience, for administrators and users, of having to
use separate computers for IDPS management and monitoring and normal production activity (using
PCs with double network interfaces that connect to both the internal and the management networks
is, of course, forbidden).

1.6 Evasion techniques

Soon after IDPS technology emerged in the corporate environment, hackers introduced several methods
for eluding detection. At first, evasion techniques were simple (DoS attacks, false positives and pattern-
matching techniques), but over time they became increasingly complex and subtle (session splicing,

7
fragmentation and polymorphic shellcode). The rest of the section will briefly outline some evasion
techniques, both simple and more complex ones.

Pattern-matching weaknesses. As previously mentioned, signature-based detection techniques

are exposed to known attacks in which the distinguishing marks have been slightly changed, so
as not to trigger an alarm in the IDPS. It is difficult to develop effective signatures that match
such obfuscations. For example, if an IDPS looks for the string GET /cgi-bin/phf?, the string GET
/cgi-bin/aaaaaaaa/..%25%2fp%68f? will not be detected as harmful, but the end host will process
it correctly. Most modern IDPSs have improved in this area, although weaknesses still exists.

Unicode evasion techniques. Unicode is an industry standard which allows computers to consis-
tently represent and manipulate text in most of the world’s writing systems (some 100,000 characters
overall). It includes several encodings, among which the widespread UTF-8 standard, which max-
imizes compatibility with the ASCII representation. The standard has been implemented in many
recent technologies, including XML, the Java programming language, the Microsoft .NET Framework
and modern operating systems. Because of some ambiguities inherent in character representations,
Unicode could be exploited to deploy attacks.
In 2000 Bruce Schneier pointed out that, with the Unicode character set, multiple representations
of a single character were possible. This is because UTF-8 went through several extensions, and each
time the representation was extended a byte, the complete previous set of characters was remapped.
Thus, for example, when UTF-8 started three-byte representations, it repeated the Unicode code
points from the two-byte representation. In addition, there are code points that may be used to
modify the previous code point. In a security context it is important to understand the meaning of a
character and these problems make the task more difficult, especially for IDPS systems. Although the
Unicode Consortium revised the Standard and made multiple representations illegal, some applications
may still be using the old standard, so IDPS signatures should still include all possible variants.
A clear example of the kind of problems that arose out of multiple representations is the Mi-
crosoft IIS 4.0 / 5.0 Extended Unicode Directory Traversal Vulnerability. IIS checked for directory
traversal before decoding UTF-8 and thus missed UTF-8-encoded directory traversals, but the oper-
ating system did not. As a result, an attempt to escape out of web root by submitting an URL like
http://myvictim/../../winnt/system32/cmd.exe was detected by IIS and an error message was
generated; but if, instead, the ../ part was UTF-8-encoded, IIS would not understand it and the
attacker could then run other programs and access other files. IDPS vendors released new signatures
for this kind of attack, but many of them would only catch a few Unicode variations of the string ../,
leaving the way open to other variants of the same attack.

Denial of service (DoS) and false positives. Many IDPSs have central logging servers that are
used solely to store IDPS alert logs. If attackers are able to find out the central logging server’s IP
address, they could slow it down or even crash it using a DoS attack. Once the server is shut down,
attacks could go unnoticed because alert data is no longer being logged. This problem can be solved
by isolating the network that IDPS components use to communicate from the organisation’s internal
network.
Another way to prevent logging servers from recording alert data and enable attacks to go unnoticed
is to fill their disk space by sending false positives. This technique could also be exploited to generate
a lot of “log noise”, by forcing IDPSs to generate a large number of false reports. This could make
the task of differentiating between false positives and real attacks very challenging.

Session splicing. The idea behind this technique is to split data between several packets, making
sure that no single packet matches any IDPS signature. It exploits how some IDPS do not reassemble
packets before performing pattern matching on data or stop reassembling after a certain delay has
elapsed between two consecutive packets belonging to the same stream. Many modern IDPSs can
nowadays handle stream reassembling, substantially putting an end to session splicing.

8
Fragmentation. Fragmentation attacks are similar to session splicing, in that attackers send packets
which do not individually trigger any alarm. However, fragmentation is generally more powerful
than session splicing. Two methods are commonly used to evade IDPS detection. The first method
overwrites part of a previous fragment, while the second one replaces a complete fragment. This
enables attackers to write entire packets of garbage, which are harmless from an IDPS’s point of
view, and craft their attack to blend in with standard protocols. Some IDPSs have found a way to
handle these types of attacks through reassembly; however, this may require in-depth knowledge of
the operating system in use on the target host and of how it implements the TCP/IP stack.

Time-To-Live attacks. Yet another way to bypass intrusion detection, time-to-live attacks require
a somewhat accurate knowledge of the internal network topology. If the attackers know the distance
(in terms of hops) to the end host, they could send packets which would only reach the IDPS sensor
by setting small TTL values, while packets with larger TTL values would be sure to reach the end
host. This would enable to inject garbage packets in the stream received by the IDPS and confuse
it, while the end host would still receive a clean packet stream. This technique can be countered by
changing the TTL flag to larger values in every received packet. If the IDPS additionally collects and
monitors TTL information for each host in the local network, further information on possible attacks
could be inferred.

Invalid RST packets. The TCP protocol uses checksums to ensure that communication is reliable.
Every time a packet is received, the end host computes the checksum and compares it to the received
one; if they differ, the packet is dropped. Attackers may exploit this feature to send RST packets with
an invalid checksum. The end host will verify the checksum value and discard the packet, but some
IDPSs may interpret the packet as an actual termination of the communication and stop reassembling
packets. However, the end host would continue accepting packets from the attacker, confusing the
IDPS.

Polymorphic and ASCII shellcode. A shellcode is a relocatable piece of machine code used as
a payload in the exploitation of a software bug. Since intrusion detection can detect signatures of
simple shellcode, the latter is often disguised through simple encoding or encryption and contains
a stub that decodes the shellcode that follows. This is known as polymorphic shellcode and hides
the commonly used strings within shellcode, which can be completely different each time it is sent,
making shellcode signatures useless. ASCII shellcode is similar to polymorphic shellcode; it contains
only valid ASCII characters, which allows attackers to bypass enforced character restrictions within
input strings. Strings are hidden within the ASCII shellcode, making pattern matching very difficult.
Using ASCII for shellcode is very restrictive, because not all Assembly instructions convert directly
to valid ASCII values. This restriction can be bypassed using other instructions or a combination of
instructions which map to ASCII and serve the same purpose.

9
Chapter 2

Types of IDPS technologies

The four main types of IDPS technologies – network-based, wireless, NBA and host-based – will
be discussed in sections 2.1 through 2.4. Honey pots, which are not IDPS technologies, but can
nonetheless be associated with them, are briefly outlined in section 2.5.

2.1 Network-based IDPS

A network-based IDPS (NIDPS) monitors wired network traffic for particular network segments or
devices and analyzes network, transport and application protocols to identify suspicious activity. Most
of the analysis is done at the application layer. Transport and network layers are also analyzed to
identify attacks at those layers and to help analysis at the application layer (e.g., a TCP port number
could indicate which application is being used). The hardware-layer is sometimes analyzed to a limited
extent.

2.1.1 Components and architectures

A typical network-based IDPS includes all the basic components of an IDPS. Whenever feasible,
these components should be connected through a management network. The sensors to be used with
NIDPS solutions are equipped with network interfaces placed into promiscuous mode, which allows
them to accept all packets, regardless of their intended destination. Sensors are available in two
formats: appliance and software-only. An appliance-based sensor is a piece of specialized hardware
which comprises NICs optimized for efficient capture of traffic; it also includes specialized processors
or other hardware components that assist in analysis, as well as sensor software, which might reside
in firmware for increased efficiency. It often uses a special operating system that administrators are
not supposed to access directly. A software-only sensor is a piece of software which is intended for use
with normal hosts, provided that they meet certain requirements. It can be either in the form of a
customized OS or a normal application that runs onto a standard OS.
Sensors can be deployed in one of two modes: inline or passive. This choice influences the preven-
tion capabilities of the IDPS solution, as shall be discussed in section 2.1.3.

Inline sensors. An inline sensor is located so that the network traffic it is intended to monitor must
pass through it. The main reason for deploying inline sensors is to enable them to stop attacks by
blocking traffic. They are usually placed where firewalls and other network security devices would
be placed, i.e. at the boundaries between networks. Some inline sensors maybe integrated into an
existing firewall device. If this is not the case, inline sensors are preferably located on the more secure
side of a network division; for example, if a firewall is present at the boundary between an external
and an internal network, the sensors should be placed on the internal side, so that they have less
traffic to process. Depending on the circumstances, they might also be placed on the less secure side
of the division to provide protection for and reduce the load on the dividing device. Figure 2.1 shows
an inline sensor deployment.

10
 Figure 2.1: Network-based IDPS sensors: an inline deployment.

Passive sensors. A passive sensor, on the other hand, is deployed in such a way as to monitor a
copy of the network traffic, but no traffic actually passes through it. Passive sensors may be used
to monitor key network segments, such as activity on a demilitarized zone (DMZ) subnet. A typical
deployment is shown in Figure 2.2.
There are several means to passively monitor traffic; the main ones are spanning ports, hubs,
network taps and IDS load balancers.
A spanning port, also known as SPAN (Switch Port ANalyzer) port, is commonly available on
switches and allows to see all traffic that passes through the switch. It works by configuring the
switch to copy trasmitted traffic or received traffic (or both) from one port or VLAN to another
port. It is easy and inexpensive to use and, since most switches support spanning ports, it does not
require additional hardare; also, it does not impact the functioning of other devices, such as firewalls.
However, it can lead to some problems. If the switch is not correctly configured or under heavy load,
the spanning port may not see all traffic. This is especially true when multiple ports are spanned to
a single port; although possible, this can quickly overload the span port. Moreover, in case of heavy
load, spanning might be temporarily disabled. Finally, most switches only have one spanning port,
which may not be enough if multiple network monitoring tools are used.
A hub may also be used to passively monitor traffic. It is placed between the connections to be
monitored, for example a router and a switch. This allows traffic to still flow between them, while the
properties of the hub cause a copy of the traffic to be copied off to the IDPS. Despite its simplicity
and reduced cost (4-port hubs are very economical), the use of hubs is not recommended. Being a
layer-1 device, a hub is inadequate when the connection between the switch and the router is a full-
duplex connection, as collisions would degrade the throughput. Similarly, it should not be used when
the IDPS sensor communicates with the other IDPS components through the company’s standard
networks, since collisions would impact the flow of traffic between the router and the switch.
A network tap represents a further solution, indeed very similar to a hub. It creates a direct
connection between a sensor and the physical network, usually a fiber optic cable, and provides the

11
 Figure 2.2: Network-based IDPS sensors: a passive deployment.

sensor with a copy of all network traffic being carried by the media. Unlike hubs, which are prone
to failure (especially low-cost ones), taps are by design fault-tolerant, having the main connection
hard-wired into the device. They do not impact the traffic flow and, once a tap is in place, changes
to the IDPS infrastructure do not influence the overall network. Disadvantages to using taps are that
they must be purchased separately and installed, which generally causes some network downtime.
Finally, an IDS load balancer is a device that aggregates traffic from one or more spanning ports
or network taps and directs it to monitoring devices, such as IDPS sensors. The volume of traffic may
be entirely sent to each sensor or it may be split among multiple sensors to balance the load (hence
the name) and avoid overwhelming; some load balancers can also be configured to direct traffic to
different devices based on IP addresses, protocols and other characteristics, so that specific analysis
can be done on certain types of traffic. Splitting traffic among multiple sensors may reduce detection
accuracy, because related events or portions of a single attack may be seen by different sensors.

2.1.2 Detection capabilities

Network-based IDPSs typically make use of a combination of the three detection techniques formerly
discussed (signature-based, anomaly-based and stateful protocol analysis) in a tightly interwoven way.
For example, stateful protocol analysis might be used to decompose traffic into requests-responses
pairs and each of them may be examined for anomalies and compared to signatures.
They can usually detect reconnaissance activity and potential threats at the application, transport
and network layers by analyzing a number of protocols: DHCP, DNS, FTP, HTTP, POP, IMAP,
SMTP, IRC, peer-to-peer file sharing protocols, and many others, at the application layer; TCP and
UDP at the transport layer; IPv4, ICMP and IGMP at the network layer. Many products are also
adding support for IPv6, which can range from very simple forms (such as identifying that IPv6 traffic
is present) to full analysis capabilities. Unexpected services, such as tunnelled protocols, backdoors,
host running unauthorized application services, are detected through stateful protocol analysis, which
can determine if the activity in a connection is consistent with the expected application protocol, or
through anomaly detection methods, which can identify changes in network flows and open ports on
hosts. Policy violations can be identified by looking at port numbers, IP addresses and website names.

12
 Some IDPSs are able to identify which operating system is running on each host through various
techniques. For example, the ports used by a host or certain characteristics in packet headers can
indicate that a particular OS or an OS family is being used (e.g., Windows, Unix). The same class
of techniques, known as passive fingerprinting, can be applied to the application layer to infer which
applications (web browsers, web servers etc.), and sometimes which version, are running on a host.
Such information may be proficiently exploited by the IDPS to understand whether a given attack is
likely to succeed or not, because for example the targeted application has a known vulnerability, and
generate high- or low-priority alerts accordingly (attacks are typically stopped in any case, whether
or not they could easily succeed).

2.1.3 Prevention capabilities

Depending on the type of sensor deployment, an IDPS solution may offer the following prevention
capabilities:
• Ending the current TCP session (passive sensors). This is done by sending TCP reset
packets to both endpoint, to make it appear to each endpoint that the other endpoint is trying
to end the connection. It is sometimes called session sniping. In many cases this technique is
ineffective because the reset packets are not received in time. Also, it is only applicable to TCP
and therefore cannot stop attacks carried in other protocols.

• Performing inline firewalling (inline sensors), which enables the sensor to drop or reject
suspicious network activity.

• Throttling network usage (inline sensors). It enables to limit the percentage of network
bandwidth consumed by a specific protocol, if it is being used inappropriately. This prevents
the activity from negatively impacting bandwidth usage for other resources.

• Altering malicious content (inline sensors). Some IDPS sensors can identify malicious content
in a packet and replace it with benign content. Such sanitization may be performed on headers,
to block some attacks involving packet headers or application headers or, additionally, it can
strip infected attachments from e-mails, as well as other pieces of malicious content.

• Reconfiguring other network security devices (both passive and inline sensors). This
prevention technique instructs network security devices such as firewalls, routers and switches
to reconfigure themselves to block certain types of activity or route it somewhere else.

• Running a third-party program or script (both passive and inline sensors). Some IDPS
sensors can run a specified program or script when certain kind of alerts are raised. Third-party
programs and scripts are most commonly used when the IDPS does not support the desired
prevention action(s).

2.1.4 Detection accuracy and tuning

Historically, network-based IDPSs have been associated with high rates of false positives and false
negatives. This is because most of them relied on signature-based techniques; nowadays, the adoption
of multiple detection strategies has provided better accuracy.
Nonetheless, the number and variety of operating systems, applications and protocols that interact
over the monitored network can be large and complex, making it impossible for a sensor to understand
everything it sees. Stateful protocol analysis techniques often attempt to replicate the processing of
packets and connections that would be done on the endpoint, slightly improving accuracy.
Considerable tuning and customization are required to achieve acceptable results. Administrators
can specify thresholds for port scans and application authentication attempts, blacklists and whitelists
for host IP addresses and usernames. Information regarding the different hosts can be used to refine
detection. For example, the IDPS could be instructed on which addresses correspond to a given type
of service, such as a web server or a mail server, which version of a given application runs on a host,
etc.

13
2.1.5 Limitations
Network-based IDPSs have some significant limitations. Three particularly important ones relate to
how the system deals with encrypted traffic, high traffic loads and attacks against the IDPS itself.
NIDPSs cannot detect attacks within encrypted network traffic, including Virtual Private Network
(VPN) connections, HTTP over SSL (HTTPS) and SSH sessions. Some of them are able to perform
an analysis of the initial negotiation conducted when establishing encrypted communications, in order
to identify known vulnerabilities or misconfiguration. Encrypted payloads should be analysed before
encryption or after decryption, which is possible with traffic that entered an organization through a
VPN gateway and has since been decrypted, but impossible in most other cases. A host-based IDPS
solution would better suit this need.
Under high loads, a network-based IDPS may not be able to perform a full analysis of the traffic
and have to drop some packets. For passive IDPS sensors this would result in some incidents going
undetected, especially if stateful protocol techniques are in use. For inline sensors, dropping packets
would cause disruptions in network availability and delays in processing packets could lead to unac-
ceptably high latency. Many implementations are able to detect high-load conditions and either pass
certain types of traffic without performing a full scan or drop low-priority traffic. Vendors typically
rate their sensors by maximum bandwidth capacity, but the actual capacity of any product is a func-
tion of several parameters, including the protocols in use, the depth of analysis performed for each
protocol, the longevity of connections and the number of simultaneous connections.
Large volumes of traffic may also indicate distributed denial of service (DDoS) attacks and anoma-
lous activity, such as fragmented packets, intended to exhaust a sensor’s resources or cause it to crash.
Other types of attack, known as blinding, could generate traffic that is specially crafted to trigger
many alerts in a short period of time, but usually not intended to perform any actual attack. The
real attack is run separately at the same as the blinding traffic, in the hope that the IDPS will fail to
detect it or that its alert will go unnoticed. Some IDPS sensors, however, are now able to recognize
common DDoS and blinding techniques and can ignore the rest of the activity, reducing the load on
the sensors.

2.2 Wireless IDPS

A wireless IDPS monitors wireless network traffic and analyzes its wireless networking protocols to
identify suspicious activity. The threats that wireless networks and a wired networks have to face
are generally the same, but the relative risk of some threats varies significantly. Wireless attacks
typically require the attacker to be within close proximity to the wireless network, whereas many
attacks on wired networks can be performed remotely from any location. However, many WLANs are
configured so that they only require weak forms of authentication, if any, making the job much simpler
for local attackers. Man-in-the-middle attacks or attacks that rely on injecting additional messages
into network communications can thus be easily carried out.

2.2.1 Components and architectures

The typical components in a wireless IDPS are the same as a network-based IDPS: consoles, database
servers, management servers and sensors. Wireless sensors play the same role as NIDPS sensors,
but they function very differently because of the complexities of monitoring wireless communications.
While NIDPS sensors can see all packets on the network they monitor, wireless sensors work by
sampling traffic.
There are two frequency bands to monitor, 2.4 GHz (IEEE 802.11b, g) and 5 GHz (IEEE 802.11a),
separated into channels. It is not currently possible to monitor all channels in a band simultaneously. A
sensor can only monitor one channel at a time; when it is ready to switch to a different channel, it must
shut its radio off, change the channel and turn its radio back on. Sensors typically change channels
frequently (several times per second), which is known as channel scanning, to avoid overlooking the
other channels for long intervals and thus potentially missing out some attacks. Specialized sensors are

14
available to reduce channel scanning; they are equipped with several radios and high-power antennas
and each radio/antenna pair is used to monitor a different channel.
Wireless sensors are available in multiple forms: dedicated, bundled with an access point and
bundled with a wireless switch.
Dedicated sensors, which provide better performance than the other types, perform wireless traffic
sniffing in a passive way, without passing it from source to destination. Some of them are also capable
of analysing the traffic they monitor, while others forward it to a management server for analysis.
Two deployments are possible: fixed or mobile. A deployment with dedicated sensors is illustrated in
Figure 2.3.

Figure 2.3: A deployment of wireless-based IDPS dedicated sensors.

Some vendors have added IDPS capabilities to access points. Such devices divide their time between
providing network access and monitoring traffic, which inevitably results in a less rigorous detection
capability than a dedicated sensor. If multiple channels or bands need to be scanned, the basic AP
functionality will be disrupted, which ultimately makes bundled sensors a less efficient solution.
Wireless switches, which are intended to assist administrators with managing and monitoring
wireless devices, can also integrate some wireless IDPS capabilities, typically not as strong as bundled
APs or dedicated sensors.
Wireless IDPS components are typically connected to each other through a wired network; because
there should already be a strictly controlled separation between the wireless and wired networks, using
either a management network or a standard network should be acceptable. Sensor locations should be
chosen so that sensors can monitor the RF range of the organisation’s WLANs. Many organisations
also want to deploy sensors to monitor physical regions where no WLAN activity should be present,
as well as bands and channels that the organisation’s WLANs should not use, as a way of detecting
rogue APs and ad-hoc WLANs.

2.2.2 Detection capabilities

Wireless IDPSs can detect attacks, misconfigurations and policy violations at the WLAN protocol
level, primarily examining IEEE 802.11a, b, g and i protocol communication. Higher-level protocols
are not taken into account. Some wireless IDPS products only use simple signature-based detection,

15
while others include several detection techniques to achieve broader and more accurate detection.
Wireless sensors can detect a number of events, such as unauthorized WLANs and WLAN devices
and poorly secured WLAN devices. This is accomplished by identifying deviations from organisation-
specific policies for settings such as encryption, authentication, data rates, SSID names and channels.
For example, a sensor could detect that WEP is being used instead of WPA 2. This category includes
the majority of the events that are ever detected by wireless IDPSs.
Some sensors can also use anomaly-based methods to detect unusual WLAN usage patterns, such
as irregular amounts of traffic, several failed attempts to join the WLAN or unauthorized WLAN
activity during off-hours periods. The use of active wireless network scanners, which aim at identifying
unsecured or weakly secured WLANs, can also be detected by some sensors, as well as spoofing the
identity of another device. DoS attacks on wireless networks include logical attacks such as flooding,
which involve sending large quantities of messages, and physical attacks such as jamming, which emits
electromagnetic energy on the WLAN’s frequencies in order to make it unusable. This kind of attacks
can be detected through stateful protocol analysis or by counting events such as session terminations
over a period of time and alerting when threshold values are exceeded.
Once a threat has been detected, most IDPS sensors can identify its location by using triangulation.
If IDPS devices can be configured to include information on the organisation’s buildings, sensors might
also be able to identify whether a threat is inside or outside the organisation’s facilities.

2.2.3 Prevention capabilities

Wireless IDPS prevention capabilities may be split into two categories: wireless and wired.
Wireless prevention actions are performed by means of the sensor itself. Some sensors can terminate
connections between a rogue or misconfigured station and an authorized access point or between an
authorized station and a rogue or misconfigured access point. This is done by sending messages to the
endpoints, telling them to terminate the current association. When a sensor is transmitting signals
to terminate connections, it may stop its monitoring activity until the prevention action has been
completed; to avoid this, some sensors have two radios, one for detection and one for prevention.
Wired prevention, on the other hand, is done by instructing a switch on the wired network to block
all activity concerning a given station or access point, based on its MAC address or the switch port it
is connected to. This technique only stops the malicious device from communicating with the wired
network, but it doesn’t prevent it from pursuing its action through the air.

2.2.4 Detection accuracy and tuning

Compared to other forms of IDPS, wireless IDPS is usually more accurate, due to its characteristics
which limit the scope of analysis to wireless networking protocols. For the same reason, not many tun-
ings and customizations are available. False positives can be generated by anomaly-based techniques if
threshold values are not properly set. Most wireless IDPSs support blacklists and whitelists of known
malicious and benign WLAN devices, which can also be used to record authorized or unauthorized
WLAN NIC vendors.

2.2.5 Limitations
While offering rather robust detection capabilities, wireless IDPSs still suffer some limitations.
The first type of threat to which wireless IDPSs are exposed is passive monitoring of traffic, i.e.
monitoring activity over the wireless network without generating any traffic, which cannot be detected
by sensors. The attacker can therefore capture traffic unnoticed, process it offline and, if some weak
protection is being used (such as WEP), he can find the encryption key used to provide security for
the wireless traffic without much effort. With this key, the attacker can decrypt all traffic from the
WLAN. An IDPS cannot fully compensate for the use of insecure wireless networking protocols.
Another problem with some wireless IDPS sensors is the use of evasion techniques. Attackers might
resort to fingerprinting to infer which IDPS product is in use. Once the product has been identified,
evasion techniques can be used to take advantage of the product’s channel scanning scheme. For
instance, attacks could be performed in very short bursts on channels that are not currently being

16
monitored or on two different channels at the same time, so that only one can be detected. A further
disadvantage of channel scanning, although not related to evasion, is that each sensor only sees a
fraction of the activity on each channel, making it considerably more difficult to analyze.
Wireless IDPS sensors are also susceptible to attack; the same attacks (logical and physical DoS)
that aim at disrupting WLANs can also disrupt sensor functions. They are also easily noticeable and
hence vulnerable to physical damage; for that reason, they are often designed to look like fire alarms
or regular access points.

2.3 Network Behavior Analysis (NBA) system

A flow is a particular communication session occurring between hosts. A network behaviour analysis
(NBA) system examines network traffic or statistics on network traffic to identify unusual flows, such
as distributed denial of service attacks, certain forms of malware and policy violations. The difference
with a network-based IDPS solution does not seem striking: they serve much the same purposes, with
NBA systems making broader use of anomaly-based techniques.

2.3.1 Components and architecture

NBA solutions typically have sensors and consoles; some products also include management servers,
which are sometimes called analyzers. Sensors usually come in the form of appliances. Some of them
are similar to NIDPS sensors and sniff traffic directly from one or more network segments; others do
not monitor the networks directly, but instead rely on network flow information provided by routers
and other networking devices. Many standards exist for flow data formats, such as NetFlow and
sFlow.
Typical flow data particularly relevant to intrusion detection and prevention includes source and
destination IP addresses, source and destination TCP and UDP ports, ICMP types and codes, number
of packets and number of bytes transmitted in the session, timestamps for the start and end of the
session.
A separate management network may be used to interconnect the various components; if sensors
collect flow information from other devices, rather than from the networks themselves, the NBA
solution can be logically separated from the rest of the organisation’s networks. An example of an
NBA network architecture is given in Figure 2.4.
Sensors are usually deployed in passive mode, using one of the connection methods outlined for
network-based IDPS (network tap, spanning port), and should be placed to monitor key network
locations, such as the divisions between networks, or key network segments, such as DMZ subnets.

2.3.2 Detection and prevention capabilities

NBA technologies can detect several types of malicious activity. Most of them use primarily anomaly-
based detection, along with some stateful protocol analysis techniques; signature-based detection is
usually not available. The types of threat that can be identified include: denial of service attacks,
by noticing significantly increased bandwidth usage or unusual data flows to/from specific hosts;
scanning, which generates atypical flow patterns; worms, which may generate large amounts of traffic,
cause hosts to communicate with each other or use ports that are normally not used; unexpected
application services, detected through stateful protocol analysis methods; policy violations. Most
NBA systems can also reconstruct a series of events to determine the origin of a threat, such as
identify the host on the organisation’s network that first transmitted a worm to the others.
The prevention capabilities offered by NBA solutions are not substantially different from those
provided by network-based IDPSs. Since sensors are typically deployed passively, prevention actions
range from session sniping to reconfiguring other network devices or running third-party scripts or
programs.

17
 Figure 2.4: An NBA IDPS deployment.

2.3.3 Detection accuracy and tuning

NBA solutions are strongest at detecting significant deviations from normal behaviour, such as un-
usually large amounts of traffic or anomalous flow patterns. They are less accurate at identifying
small-scale attacks, especially if they are conducted slowly. NBA sensors can take some time to label
an activity detected on the networks as abnormal behaviour, so many attacks do not trigger any
alarm until they reach a point where their activity is significantly different from what is expected.
As a consequence, NBA systems may not be the ideal solution when a timely response to attacks is
fundamental.
Sensors can usually be tuned and made more sensitive to anomalous activity, which means alerts
will be generated more promptly when attacks occur; however, this will increase the number of false
positives. The latter can also be caused by benign changes in the environment, such as adding a new
service or moving an existing service to a different host.
Besides sensor sensitivity, not much tuning is usually required. NBA products automatically
update their inventories of expected flows and host characteristics on an ongoing basis. Administrators
might periodically adjust thresholds, update firewall ruleset-like policies and whitelists and blacklists
of hosts and services.

2.3.4 Limitations
As mentioned in the previous section, the main limitation to NBA solutions is the delay in detecting
attacks. Some delays are inherent in anomaly detection methods which, by their own nature, need
time to assess purported deviations from normal behaviour. However, NBA technologies often suffer
from additional delays caused by their data sources, especially when sensors rely on flow data from
routers and other networking devices. As a matter of fact, this data is often transferred to the NBA
system in batches; depending on the product and on its configuration, this could occur relatively
frequently (e.g., every minute) or relatively infrequently (e.g., every 15 minutes). Chances are, when
long delays are present, that an attack be detected once it has already disrupted or damaged systems.

18
2.4 Host-based IDPS
Unlike other types of IDPS solutions, which are intended to monitor all traffic on a network, re-
gardless of its source, a host-based IDPS monitors the characteristics of a single host and the events
occurring within that host for suspicious activity. A host-based IDPS could for example monitor
incoming and outgoing wired and wireless network traffic, system logs, running processes, file access
and modification, system and application configuration changes.
The same kind of monitoring is often provided by a number of anti-virus software and personal
firewall solutions. The distinction between these tools becomes very blurred, as many of them overlap
in functionality.

2.4.1 Components and architecture

Host-based IDPSs include agents, which are the equivalents of sensors and transmit data to manage-
ment servers, which in turn may optionally use database servers for storage. Consoles are also used for
management and monitoring. In some cases, agents are software-only; this means they are installed
on the host they guard as any other application and interact with the operating system. Alternatively,
some products use dedicated appliances running agent software instead of installing agent software
on individual hosts. In that case, each appliance is positioned to monitor traffic going to and from
a particular host. These appliances are very similar to network-based IDPS devices, but they usu-
ally offer monitoring capabilities for only one specific type of application, such as a web server or a
database server, so they are more specialized than standard network-based IDPSs. Advantages and
disadvantages to each type of agent will be discussed later.
Agents are typically designed to protect server or client hosts or, sometimes, they can perform
monitoring for a specific application service only. This type of agent is also known as an application-
based IDPS. Other types of network devices, such as firewalls, routers and switches, cannot normally
include a host-based agent.
The network architecture for host-based IDPS deployment is rather simple. The components
usually communicate by means of the organization’s existing networks instead of using a separate
management network. This is to avoid having to equip the guarded hosts with multiple network
interfaces and to ensure that traffic cannot pass from one interface to the other. Most products encrypt
their communications. Appliance-based agents are typically deployed inline immediately in front of
the hosts that they are protecting. Figure 2.5 shows an example of a host-based IDPS deployment
architecture.
Host-based IDPS agents are most commonly deployed to critical hosts, such as publicly accessible
servers and servers that contain sensitive information. Agents being available for a number of server
and desktop/laptop operating systems, some organizations could decide to install them on most of
their hosts.
In addition, host-based IDPS agents are often used because they can analyze activity that cannot
be monitored by other security controls; an example of that is activity within encrypted network
communications, which cannot be analyzed by common network-based sensors. Host-based agents,
on the other hand, are installed on endpoints and can analyse traffic before encryption and after
decryption.

2.4.2 Host agents architecture

Most software agents alter the internal architecture of the hosts they monitor in order to provide
intrusion prevention capabilities. This is typically done through a shim, which is a layer of code
interposed between existing layers of code. A shim intercepts data at a point where it would normally
be passed from one piece of code to another; it can analyze it and determine if the action should
be permitted or not. Shims are used to control several types of resources, such as network traffic,
filesystem activity, system calls, Windows registry activity and common applications.
The use of shims enables the IDPS to have a rather accurate overview of the activity occurring on
the host. Nevertheless, intercepting and analysing all activity on a host can be very resource intensive

19
 Figure 2.5: A host-based IDPS deployment.

and add a significant strain on the system, negatively impacting its performance. For that reason,
some host-based IDPS do not alter the host architecture; instead, they monitor activity without
shims, by analysing application log entries and file modifications. Although less intrusive and more
lightweight to the host, these methods are also generally less effective at detecting threats and often
cannot perform many prevention actions.
The principle of operation of a host-based agent depends on the fact that successful intruders
will generally leave a trace of their activities. Such intruders will often establish their ownership by
installing software that will grant them future access to carry out whatever activity the envisage, such
as keystroke logging, identity theft, spamming, botnet activity, spyware-usage. These traces could be
identified by a skilled computer user; the IDPS automates such a task.
In general, the agent will create a database of system objects it should monitor, usually file-system
objects, and check that particular regions of memory have not been modified, like the system-call table.
For each object at stake the agent will remember its attributes (permissions, sizes, modification dates)
and create a checksum for the contents (MD5, SHA-1 hash or similar). This information will then be
stored in a database and secured. Since the process of initialising the checksum database generally
requires a long time and involves cryptographically locking each monitored object, manufacturers
usually design the object database in such a way that frequent updates are not necessary. Computer
systems also have many dynamic, frequently changing objects, whose nature makes them unsuitable
for checksum techniques; a raft of other means are then used to monitor changes.
Once the object and checksum databases have been properly constructed, the agent regularly
scans the monitored objects, reports on anything that may appear suspicious and stops potentially
dangerous actions. Since intrusions might still succeed, the IDPS databases are just as likely to be
tampered as the objects they guard. For that reason, some IDPSs might allow administrators to store
the databases on a read-only memory device, such as a cd-rom. Similarly, logs produced by the IDPS
are often sent off-system immediately, sometimes via one-way communication channels.

20
2.4.3 Detection and prevention capabilities
Most host-based IDPSs use a combination of signature- and anomaly-based detection techniques.
Specific techniques that are commonly used include:

• Code analysis. These techniques aim at identifying malicious activity by analysing attempts
to execute code. For example, code behaviour analysis can first execute code in a virtual en-
vironment and compare its behaviour to profiles or rules; buffer overflow detection identifies
typical sequences of instructions that attempt to perform stack and heap buffer overflows; other
techniques are able to monitor system calls and restrict which drivers and dynamic link libraries
(DLL) can be loaded.

• Network traffic analysis and filtering. This is often similar to what a network-based IDPS
does. It analyses network, transport and application layer protocols and includes processing for
common applications. Agents often include a host-based firewall that can restrict incoming and
outgoing traffic for the system.

• Filesystem monitoring. It includes a number of methods, such as file integrity checking, which
generates checksums for critical files and periodically verifies them, and file attribute checking;
these two methods can only determine after-the-fact that a file has been changed. Some agents,
typically those which use a shim, are able to monitor all attempts to access critical files and
stop attempts that are suspicious. The current attempt is compared against a set of policies
regarding file access and blocked if the type of access that has been requested (read, write,
execute) contradicts a policy.

• Log analysis. Some agents can identify malicious activity by monitoring and analyzing system
and application logs, which contain information on events such as shutting down the system,
starting a service, as well as application startup and shutdown, failures, major configuration
changes.

• Network configuration monitoring. Agents are able to monitor a host’s current network
configuration and detect changes to it. For example, network interfaces being placed in promis-
cuous mode, additional TCP or UDP ports or unusual protocols being used (such as non-IP
protocols) could indicate that the host has already been compromised and is being configured
for use in future attacks or for transferring data.

• Removable media restriction. Some products can enforce restrictions on the use of removable
media, which can prevent malicious content from being transferred to a host and stop sensitive
files from being copied from the host.

• Process status monitoring. Some host-based IDPSs can monitor the status of the processes
and services running on a host; when they detect that one has stopped, they restart it auto-
matically. This provides protection against some forms of malware which can sometimes disable
antivirus software and the like.

• Network traffic sanitization. This form of protection is usually implemented by appliance-

based IDPSs. Sanitization of traffic may rebuild all requests and responses directed to the host
or coming from it, thus neutralizing certain unusual activity, particularly in packet headers and
application protocol headers. It can also reduce the amount of reconnaissance the attackers can
perform on the host, by hiding OS fingerprints and application error messages.

2.4.4 Detection accuracy and tuning

The accuracy of detection for host-based IDPS solutions is generally more challenging than for the
other types of IDPSs. While the events occurring on the guarded host are detected rather accurately,
their benign or malicious nature is often hard to determine. This is due to the fact that agents
sometimes lack sufficient context information. Some of them, particularly those intended for desktop

21
or laptop use, prompt users to provide context, such as whether or not the user intentionally started
an update or made a configuration change.
Considerable tuning may be required to improve detection accuracy. Some agents which make use
of anomaly-based techniques might need to observe host activity during a certain period of time in
order to develop profiles of expected behaviour. Others need to be configured with detailed policies
which define, on a per-application basis, which operations should be permitted and what should be
considered as normal behaviour. Other customization capabilities include the definition of blacklists
and whitelists for hosts, applications, ports, filenames and other objects.

2.4.5 Limitations
Host-based IDPS have significant accuracy limitations, as has already been considered. Another
drawback is represented by the fact that, unlike the other IDPS technologies, with host-based IDPSs
the guarded system and the guardian coincide. This means that agents consume the host’s resources,
such as memory, processor usage and disk storage, especially if they make use of shims. When this
would slow down performance in an unacceptable way, IDPS functionalities can be delegated to an
appliance-based agent. This solution, despite leaving the host’s resources free, usually provides less
accurate and in-depth detection.
Delays in generating alerts are another weakness of host-based IDPSs. This may be due to some
detection techniques which only scan the system for threats a few times a day. What is more, many
agents are intended to forward their alert data to the management servers on a periodic basis and not
in real-time. For example, alert data may be transferred in batches every 15 to 60 minutes to reduce
overhead for the IDPS components and the network. This can cause delays in initiating response
actions, which increases the impact of incidents that spread quickly, such as malware infestations.

2.5 Honey pots

As outlined in section 1.1, one of the purposes of IDPSs is to gather information relative to the
types of threats an organisation receives and to study attackers’ techniques and aims. Once sufficient
information has been acquired, it can be used to harden security on the company’s servers. A way of
collecting such information without compromising the assets of the company is using honey pots.
As their name suggests, honey pots are systems intended to lure hackers by exposing known
vulnerabilities deliberately. They are valuable as a surveillance and early-warning tools, but they
should have no production value and hence should not get in touch with any legitimate traffic, activity
or data. It is important, however, that honey pots look like real systems and appear to be part of a
network. Fake data files, user accounts and the like should be created to reassure a hacker.
There are different ways to build and place honey pots. A honey pot should have common services
running on it, such as a Telnet server, a web server, an FTP server and so on. It could be placed
somewhat near real production servers, so that it can be confused with a real server, for example by
assigning contiguous IP addresses to them. Firewalls or routers could also be configured to redirect
traffic on some ports to a honey pot.
Although useful, a honey pot may carry risks to a network and must be handled with care. If not
properly isolated, a honey pot could be used by an attacker to break into real systems. An organisation
should deploy a honey pot only if it has the resources both to manage it and to inspect the collected
data and extract valuable information from it.

22
Chapter 3

Snort: an open-source NIDPS

Snort is a free and open-source network-based intrusion detection and prevention system. It is a
software-based NIDPS. Originally written in 1998 by Martin Roesch, today it has arguably become a
“de facto standard in (network) intrusion detection and prevention” – exactly what the product prides
itself on being. It is now developed and supported by Sourcefire, of which Roesch is the founder and
CTO.
Snort performs protocol analysis and content searching and matching; it is commonly used both
to actively block and passively detect a variety of attacks and probes.

3.1 Modes of operation

Snort can be configured to run in three modes: sniffer/packet logger mode, network intrusion detection
system mode and inline mode.
When in the sniffer mode, Snort simply reads the packets from the network and displays them in
a continuous stream on the console. The packet logger mode does the same thing, except for logging
the packets to disk.
The network intrusion detection system mode is the most complex and widely configurable mode.
It allows Snort to analyse network traffic for matches against a user-defined rule set and performs
several actions.
The inline mode is designed to obtain packets from iptables, rather than from libpcap (a library
that provides a system-independent API for low-level network monitoring). It can then cause iptables
to drop or pass packets based on Snort rules.

3.2 Components
Snort is logically divided into multiple components, which work together to detect particular attacks
and to generate output in a given format. The major components are the packet decoder, the prepro-
cessors, the detection engine, the logging and alerting system and the output modules. A brief overview
of each component is given in the paragraphs that follow. Figure 3.1 illustrates the architecture of
Snort.

Packet decoder. The packet decoder’s role is simple: it takes packets from different types of network
interfaces and prepares them to be pro-precessed or to be sent to the detection engine. The interfaces
may be Ethernet, SLIP, PPP and others.

Preprocessors. Preprocessors are components or plug-ins that can be used with Snort to arrange
or modify data packets before they are analysed by the detection engine. Some preprocessors also
perform some form of detection by identifying anomalies in packet headers and generating alarms.
Preprocessors are very important when packets are compared against a set of rules, as it is the case
with Snort. Hackers may use a number of evasion techniques to bypass detection, as explained in
section 1.6. Some of these techniques can be effectively countered by preprocessors. They can, for

23
 Figure 3.1: Architecture of the Snort NIDPS.

instance, analyse and decode strings containing hexadecimal or Unicode characters to avoid character
encoding-based evasion, parse strings that include paths, defragment packets, reassemble TCP streams
etc.

Detection engine. The detection engine is the core of the Snort IDPS. Its responsibility is to detect
if any intrusion activity exists in a packet. Snort employs sets of rules to do so; rules are written in
files by the administrator and copied to internal data structures, or chains, at load time. If a packet
matches any rule, appropriate action is taken, otherwise the packet is dropped. Appropriate actions
may be logging the packet or generating alerts.
The detection engine is the time-critical part of Snort. Depending on the power of the machine
that is running it, on how many rules have been defined and on the load of the network, it may take
different amounts of time to respond to different packets; when network traffic is too high this could
result in dropping some packets and not getting a true real-time response. Once the choice of the
machine on which to deploy Snort has been made, efforts should be made to catch as many signatures
as possible with the smallest number of rules.
The detection engine can dissect a packet and apply rules on different parts, such as the different
headers (IP, transport, application) or the packet payload. In all 1.x versions, the detection engine
would stop further processing of a packet once a rule was matched and action would be taken according
to the rule. This meant that if a packet matched criteria defined in multiple rules, only the first rule
to be matched in the chain was applied, regardless of the other rules. This behaviour represented a
security problem: if a low-priority priority rule was matched first, all subsequent rules were ignored,
even if high-priority. As of version 2, the problem has been rectified by matching all possible rules
against a packet and, only then, generating an alert, based on the highest-priority matched rule.

Logging and alerting system. Depending upon what the detection engine finds inside a packet,
the latter may be used to log the activity or generate an alert. Logs may be produced in the form of
simple text files, in tcpdump-style files or some other form. The tcpdump binary mode usually allows
higher logging speed and can be understood, for later analysis, by common sniffer programs, such as
tcpdump or Ethereal. Command line options are available to modify the type and detail of information
that is logged. Alerts can be generated in one of many ways; for example, they can be included in the
log files, sent to a UNIX socket where another program is listening or displayed in a user console.

Output modules. Output modules or plug-ins process alerts and logs and can perform different
operations. Depending on the desired type of output or response, output modules can do things as
simple as logging to the default folder, sending messages to the syslog facility, or more complex such
as logging to a database, like MySQL or Oracle, and modifiying the configuration on routers and
firewalls.

24
Conclusions

The four primary types of IDPS technologies described in this report each offer fundamentally different
information gathering, logging, detection, and prevention capabilities. Each technology type offers
benefits over the other, such as detecting some events that the others cannot, detecting some events
with significantly greater accuracy than the other technologies, and performing in-depth analysis
without significantly impacting the performance of the protected hosts. Organizations should consider
using multiple types of IDPS technologies to achieve more comprehensive and accurate detection and
prevention of malicious activity, with lower rates of false positives and false negatives.
Intrusion detection systems are evolving products. Research began in the mid-1980s and, since the
appearance of the first products in the mid-1990s, this area has continued to change as new research
influences the design of products. Recent trends have shown to be headed towards the integration of
intrusion detection technology within the TCP/IP stack, yielding the so called stack-based IDPSs.

25
Bibliography

[1] K. Scarfone, P. Mell – Guide to Intrusion Detection and Prevention Systems (IDPS). NIST SP
800-94, 2007.
http://csrc.nist.gov/publications/nistpubs/800-94/SP800-94.pdf

[2] C.P. Pfleeger, S.L. Pfleeger – Security in computing. Fourth Edition, 2006.

[3] W. Stallings – Cryptography and Network Security Principles and Practices. Fourth Edition, 2005.

[4] A. Brox – Signature-Based or Anomaly-Based Intrusion Detection: The Practice and Pitfalls.
http://www.scmagazine.com/asia/news/article/419779/
signature-based-anomaly-based-intrusion-detection-practice-pitfalls

[5] E. Hacker – IDS Evasion with Unicode.

http://www.securityfocus.com/infocus/1232

[6] Intrusion Detection System (IDS) Evasion. An iDefense Security Report, 2006.
http://complianceandprivacy.com/WhitePapers/iDefense-IDS-Evasion/
iDefense IDSEvasion 20060510.pdf

[7] Snort Team – Snort User’s Manual 2.8.0.

http://www.snort.org/docs/snort manual/2.8.0/snort manual.pdf

[8] B. Laing – How To Guide: Implementing a Network-Based Intrusion Detection System

http://www.snort.org/docs/iss-placement.pdf

[9] C. Viklund – Identifying Threats in a Wireless Environment. Master’s thesis, Luleå University of
Technology, Sweden.
http://epubl.ltu.se/1402-1617/2005/234/LTU-EX-05234-SE.pdf

[10] R.U. Rehman – Intrusion Detection Systems with Snort.

http://www.informit.com/content/downloads/perens/0131407333.pdf

26