Signature-based Botnet Detection and Prevention

Sunny Behal1, Amanpreet Singh Brar2, Krishan Kumar3

1
Deptt of CSE, SBECET, Ferozepur, India
2
Deptt of CSE & IT, GNDEC, Ludhiana, India
3
Deptt of CSE, SBSCET, Ferozepur, India

Abstract primary source of most of the threats used for scanning,

The Internet is used extensively for important services
such as banking, business, medicine, education, research,
stock trades, weather forecasting etc. Most of these
services must be processed in a timely manner. However
these services are delayed, degraded and sometimes
completely disrupted because of unavailability of internet.
The inherent vulnerabilities of the internet architecture
provide opportunities for a lot of attacks on its
infrastructure and services. Behind these attacks is a
large pool of compromised hosts sitting in homes, school,
business and governments around the world. These
infected systems are called bots that communicates with a
bot controller and other bots to form what is commonly
referred to as a Zombie army or Botnet. For any
organization, internal bot infections cause serious
repercussions including loss of man hours and downtime.
The average cost of such disasters runs into tens of
thousands of dollars. So there is need to defend against
such attacks. In this paper, we have analysed the
feasibility of outbound traffic i.e. extrusions, to detect and
prevent attacks caused because of botnets. As a part of
the research work, a Network-based Detection and
Prevention systems of botnets called N-EDPS has been
proposed.
(distributed) denial-of-service (DOS) activities
and direct attacks, taking place across the Internet. At the
center of these threats is a large pool of compromised
hosts sitting in homes, schools, businesses, and
governments around the world. Bot malware typically
takes advantage of system vulnerabilities and software
bugs or hacker-installed backdoors that allow malicious
code to be installed on computers without the owners’
consent or knowledge. However, all bots distinguish
themselves from the other malware forms by their ability
to establish a command and control (C&C) channel
through which they can be updated and directed. Once
collectively under the control of a C&C server, bots form
what is referred to as a botnet. The elements involved and
the sequence of commands exchanged between different
botnet elements is shown in figure 1.

Keywords: - Attacker, Bot, Botmaster, Extrusion,

Intrusion, Pear to Pear, Zombie.

1. Introduction
The Internet consists of hundreds of millions of
computers distributed all around the world. Most of the
companies, institutes, banks, businesses, and research
heavily depend on a well-working and secure computer
networks. Any incident could be critical to their routine
work. The increasing usage of interactive internet
applications in these areas has induced a rise in risks and
possibilities of misuse of computer networks. The core
objectives of information security have to be met in order
to protect the network such as confidentiality, integrity,
availability, authentication, non-repudiation. In order to
meet all of these requirements, it is essential to protect a
network against all possible threats. Over the last decade,
malicious software or malware has risen to become a
Figure 1: Working of a typical IRC-based Botnet
Firstly, a botmaster exploits the vulnerability on the
victim. Then the victim downloads the actual bot binary
and contacts the IRC server address in the executable,
including resolving the DNS name. After that the bot
joins an IRC communication channel to receive
commands from the botmaster via communication
channel. The proposed work uses the botnet life cycle
given in [1] and is depicted by the State Transition
Diagram shown in figure 2.

Figure 2: Life cycle of a Botnet
The figure 2 is not intended to provide a strict ordering of
events, but rather to capture a typical infection dialog. In
the idealized sequence of a direct-exploit bot infection
model, the bot infection begins with an external-to-
internal communication flow that may encompass bot
scanning (E1) or a direct inbound exploit (E2). When an
internal host has been successfully compromised, the
newly compromised host down- loads and instantiates a
full malicious binary instance of the bot (E3). Once the
full binary instance of the bot is retrieved and executed,
this model accommodates two potential dialog paths,
referred to as the bot Type I versus Type II split. Under
Type II bots, the infected host proceeds to C&C server
coordination (E4) before attempting self-propagation.
Under a Type I bot, the infected host immediately moves
to outbound scanning and attack propagation (E5),
representing a classic worm infection. Botnets can serve
both legitimate and illegitimate purposes as described in
[6]. One legitimate purpose is to support the operations of
IRC channels using administrative privileges on specific
individuals. Nevertheless, such goals do not meet the vast
number of bots that we have seen. The possibilities to use
botnets for criminally motivated or for destructive goals
has been categorized as DDoS Attacks [3, 15], Spamming
and Spreading Malware [16, 17], Information Leakage
[14, 15], Click Fraud [15], Identity Fraud [15], Hosting of
Illegal Software [16], Political Activism [18]. Before
discussing Botnet defense approaches, it is necessary to
highlight here that traditional security technologies such
as router access lists, firewalls, and Intrusion detection
systems which are important components of overall
security strategy, do not provide comprehensive botnet
protection by themselves because of the reasons given in
[19]. Since these security components are unable to
restrict botnet attacks, some specific defense approaches
are being deployed to combat botnet attacks. The
stumbling barrier against these attacks is that it is almost
impossible to differentiate between legitimate and attack
packets. Therefore it has become a real challenge to
defend against these attacks. The seriousness of botnet
problem and growing sophistication of attackers have led
to development of numerous defense mechanisms. These
defense mechanisms are classified into three broad
categories in Prevention [5, 6], Detection and Tracing [4,
5, 9, 11, 12] and Mitigation mechanisms [5, 8, 10].
Network-based intrusion detection systems (IDSs) and
intrusion prevention systems (IPSs) may come to mind as
the most appealing technology for detecting and
mitigating botnet threats. Traditional IDSs, whether
signature based [20, 21] or anomaly based [1, 8], typically
focus on inbound packets flows for signs of malicious
point-to-point intrusion attempts. Network IDSs have the
capacity to detect initial incoming intrusion attempts, and
the prolific frequency with which they produce such
alarms in operational networks, However, distinguishing a
successful local host infection from the daily myriad of
scans and intrusion attempts is as critical and challenging
a task as any facet of network defense [2].
Our primary contribution in this paper is (1) to introduce a
new network monitoring strategy, which focuses on
detecting and preventing malware infections (specifically
bots/botnets) through monitoring outbound traffic i.e.
extrusions only using the available signatures. (2) to
reduce the rule set of a detection and prevention system so
as to increase its efficiency under attack. (3) to utilizes the
existing open source and freely available software to
develop a network based detection and prevention system
of botnet based attacks.
The remainder of this paper is outlined as
follows.
The section-II focuses on the problem formulation and
experimental setup of N-EDPS. The section-III discusses
the various results obtained and the last section concludes
the work by highlighting the scope for future work.
2. Related Work
The proposed work utilizes the Botnet detection system
called a BotHunter proposed in paper [1]. BotHunter is a
passive network monitoring system driven by Snort. It
correlates the inbound intrusion alarms with the outbound
communication patterns that are highly indicative of
successful local host infection. The experimental results
using BotHunter are presented in a virtual and live testing
environment. BotHunter focuses on botnet detection and
its traffic. We focus on all outbound traffic generated by a
malicious source. Paper [22] focuses on the outbound
traffic with the intention to guarantee that the host will not
be used as an attack launcher or intrusion relayer to
compromise other systems. Therefore, the intended goal
using the outbound traffic is different for the mentioned
paper and the present work. Paper [22] focuses on the
prevention of further propagation of malware. In this
work, on the other hand, the outbound traffic is analysed
to get a clear indication about a successful attack. The
proposed work also utilizes the malware classifications
given by C. Lussi in paper [2]. The author in [2] uses the
concept of escalation rules with different weights. The
applied rule with the highest weight determines the
treatment of the corresponding alert. Apart from this
work, other network-based automated botnet detection
tools are in existent like Rishi [23], Strayer [24], and
BotMiner [13]. While there are few other detection tools
available which are host based like Binder [25] and
BotSwat [26]. The gaps in the existing work have been
identified and efforts are made to address some of these
gaps as part of current work
3. Methodology
The conceptual methodology used for the development of
N-EDPS has been shown in figure 3. The end product will
first monitor the network traffic of the educational
institute to capture the details of types of attacks
occurring in the network and the output will be stored in
some database or log file for future reference. Based on
the available signatures, alerts will be generated
corresponding to the various attacks occurring in the
network.
Figure 3: Methodology to develop N-EDPS
New rules/signatures will be developed and updated to the
database of detection system as well as to the database of
prevention system. The database of prevention system
will contains only those rules for which we want to drop
the incoming packets whereas the database of detection
system will contains more no. of rules for which we want
to generate alerts only. After the development of such
rules, we would be able to filter malicious traffic from
legitimate traffic resulting in Botnet free traffic. For the
development of proposed N-EDPS, a prevention system,
also known as active IDS has been used, which
investigates the traffic inline. This means that the packets
are analysed continuously and the reaction to an attack is
in real-time. The IPS blocks traffic independently without
human interaction. It aims not only at detecting, but also
at preventing an attack. In contrast, a passive IDS does
not act by itself but does only raise an alarm in case of a
supposed attack. The Network-based deployment has
been used instead of Host-based deployment. A Network-
based IPS monitors the network traffic of a particular
network whereas a Host-based IPS monitors the operating
system, applications, and the host specific network traffic.
The concept of signature-based detection and prevention
system is used which searches for known malicious
patterns in the payload whereas a behavior-based IDS,
also known as an anomaly detection system, analyses in
the first instance the traffic data. Most of the current used
IDS focus on the intrusion from outside of the network
into the monitored network. Such a detection of attacks
creates a lot of false alarms. A new approach called
extrusion detection is focusing on the traffic, whose
source address is inside of the monitored network. The
extrusion detection technique is a promising approach
because the behaviour of an infected system and the
generated traffic due to this infection is often
conspicuous. An extrusion is a clear indication of
occurred intrusion, because an extrusion only happens as
a result of a successful attack. The differentiation between
an attempted and a successful attack is the ultimate goal
of this work. Therefore, we will analyse and test the
efficiency of the extrusion detection approach. For the
development of N-EDPS, a no. of Open Source Softwares
and free Softwares has been used. This permits users to
use, change, and improve the software, and to redistribute
it in modified or unmodified forms
Table 1: Classification of signatures for N-EDPS
File Name Description No. of
Rules
E1.rules contains all External to Internal 75
Inbound Scan related rules
E2. rules contains all External to Internal 325
Inbound Exploit related rules
E3.rules contains all Internal to External 250
Binary Acquisition related rules
E4.rules contains all Internal to External C&C 370
Communication related rules
E5.rules contains all Internal to External 455
Outbound Infection Scanning rules
4. Experimental Setup
The proposed N-EDPS consists of two components. One
is the detection engine and the other one is the prevention
engine. For the development of proposed system, we have
used Bot Hunter as the detection engine and snort-inline
as the prevention engine. The system topology for the N-
EDPS in the live environment in shown in figure 4. We
placed the proposed N-EDPS between the network and
the Internet Server to monitor all the outbound traffic. It is
worth to be noted that the IDS engine works for two
conditions;
 • Condition 1: Evidence of local host infection
(E2), AND evidence of outward bot coordination
or attack propagation (E3-E5); or
• Condition 2: At least two distinct signs of
outward bot coordination or attack propagation
(E3-E5).
As for the development of the proposed N-EDPS, we
have focussed on only outbound traffic, so we will use
condition 2 for the detection of botnets in a network. And
will use only category E3, E4 and E5 rules\signatures for
the above said purpose. The N-EDPS had been deployed
in the SBSCET network for a period of three weeks and
to run the N-EDPS, the rules\ signatures have been
classified according the life cycle of the botnet shown in
figure 2 and have been stored accordingly into five files
as shown in table 1.
Figure4:Experimental Topology of N-EDPS
Figure5:
As we concentrate only on outbound traffic, we use only
E3, E4 and E5 type rules for botnet detection.
5. Results and Discussion
The N-EDPS has been run for a period of three weeks in
the live environment of SBSCET network. As shown in
figure 5, we have been able to find 42 infected computers,
65 C & C servers, 32 Egg download servers and 24 IP
addresses of outbound scanning servers which could be
used for further infection. Apart from these results, the
signatures that have been triggered in identifying the
popular viruses/worms/spyware who were using botnets
are shown in table 2. The name and type of
viruses/worms/spyware found to use botnets inside the
network, have been shown in table 3.
Table 2: Top Triggered Signatures
ET Known Russian Business Network Monitored Domain
ET ShadowServer confirmed botnet control server
ET TROJAN Down adup/Conficker A or B Worm reporting
Detected intense malware port scanning of 30 IPs
BotHunter MTC confirmed botnet control server
ET SCAN Behavioral Unusual Port NETBIOS traffic,
Potential Scan or Infection
ET TROJAN Downadup/Conficker A Worm reporting
ET TROJAN BOT - potential response
BotHunter REPO confirmed botnet control server
Detected moderate malware port scanning of 10 IPs
ET TROJAN Sality Trojan User-Agent (KUKU v3.09 exp)
ET VIRUS Sality Virus User Agent Detected (KUKU)
ET MALWARE 180solutions Spyware Reporting
ET MALWARE Hotbar Agent Partner Checkin
ET TROJAN Likely Bot Nick in IRC (USA +..)
ET MALWARE Hotbar Agent Reporting Information
ET TROJAN Pakes/Cutwall/Kobcka Update URL Detected
BACKDOOR Pushdo client communication attempt
BotHunter HTTP-based .exe Upload on backdoor port
ET POLICY PE EXE or DLL Windows file download
ET TROJAN Peed Report to Controller
ET POLICY Outbound Multiple Non-SMTP Server Emails
COMMUNITY BOT GTBot info command
A number of unusual ports have also been found (123,
137, 443, 445, 1036, 1170, 1199, 1863, 1900, 2711, 3670,
6667, 6697, 7000, 7920, 8000, 8067, 8448, 9006, 18384,
47221, 49158, 55273, 58670). It had been confirmed that
the monitored network did not provide any service
corresponding to these ports.
 Table 3: Botnets found
Conflicker A \ Downup \ Downadup \ Kido
Conflicker B \ Downup \ Downadup \ Kido
Sality trojan user agent (KUKU v3.09 exp)
Sality Virus user agent (KUKU)
180 solutions spyware
Hotbar
Pakes
Cutwall
Kobcka
Pushdo\ Pandex \ Cutwail
Peed\ Storm
variant of GT Bot
6. Conclusion and Scope for Future Work
To better understand the botnet and to stop its attacks
eventually, this research work focuses on the detection
and prevention of successful botnet attacks based on the
concept of analyzing outbound traffic i.e. Extrusions only.
As a part of the work we have proposed a signature-based
N-EDPS which examines only outbound traffic to detect
Botnet related malicious traffic using various open source
and freely available software. We run the N-EDPS for a
period of three weeks in the live environment of SBSCET
network. As a part of the run, we have been able to find a
number of infected computers, C & C servers, Egg
download servers and outbound scanning servers which
could be used for further infection. The name and type of
viruses/worms/spyware found to use botnets inside the
network have been found. The proposed N-EDPS is better
than an N-IDPS because N-EDPS requires a smaller
database of rules \ signatures as compared to N-IDPS and
delivers better results. But there are certain drawbacks of
the proposed system. (1) As the case with any Antivirus
software, the proposed signature-based N-EDPS also
requires access to a current database of attack signatures.
(2) The proposed N-EDPS is not capable of detecting
encrypted C & C channels, if they exist. For this, anomaly
based detection logic must be incorporated into N-EDPS.
This work opens up a number of avenues for future work.
(1) A perfect N-EDPS is one which can respond to attacks
when they occur, i.e., the one which is able to provide
real-time response to any kind of attack whether it is
known or novice. For this we need to develop an N-EDPS
which integrates the feature of both signature-based and
behaviour-based detection system and dynamically
develop new rules for novel attacks and drop the traffic in
real time. (2) For detecting encrypted C & C channels, the
anomaly based logic can be incorporated.
References
[1] Gu, G., Porras, Ph., Yegneswaran, V., Fong, M., Lee, W.
“BotHunter: Detecting malware infection through IDS-
driven dialog correlation” , In 16th USENIX Security
Symposium (Security’ 07), 2007.
[2] Cecile Lussi. Master’s thesis on “Signature-based
Extrusion detection” ETHZ (TIK), 2008.
[3] F. Freiling , T. Holz and G. Wicherski , “Botnet tracking:
exploring a root-cause methodology to prevent distributed
denial-of-service Attacks,” in Proceedings of the 10th
European Symposium on Research in Computer Security
(ESORICS ’05), vol. 3679 of Lecture Notes in Computer
Science, pp. 319–335,Springer, Milan, Italy, 2005.
[4] G. Carl, G. Kesidis, “Denial of Service Attack-Detection
Techniques,” IEEE Internet Computing, Vol 10, No. 1, pp
82-89, 2006.
[5] E. Cooke, F. Jahanian , and D. McPherson , “The zombie
roundup: Understanding, detecting, and disrupting Botnets”
in Proceedings of Workshop on Steps to Reducing
Unwanted Traffic on the Internet (SRUTI’05), 2005.
[6] McCarty B. (2003), “Botnets: big and bigger,” IEEE
Security and Privacy, vol. 1, no. 4, pp. 87–90.
[7] R. Villamarin-Salomon and J. , “Identifying botnets using
anomaly detection techniques applied to DNS Traffic” in
Proceedings of the 5th IEEE Consumer Communications
and Networking Conference, pp. 476–481, Las Vegas, Nev,
USA, 2008.
[8] J. Liu, Y. Xiao, J. Zhang , ”Botnet: Classification, attacks,
Detection, Tracing and Preventive measures” EURASIP
journal of Wireless Communications and Networking, Vol.
2009, article ID 692654, 2009.
[9] Y. Kugisaki , Y. Kasahara , Y. Hori , and K. Sakurai , “Bot
detection based on Traffic Analysis,” in Proceedings of the
International Conference on Intelligent Pervasive
Computing (IPC ’07), pp. 303–306, Jeju Island, South
Korea, 2007.
[10] T. Holz , M. Steiner, F. Dahl, E. Biersack, Freiling ,
”Measurements and Mitigation of Peer-to-Peer-based
Botnets: A Case Study on StormWorm”, 2007
[11] X. Hu , M. Knyz and K. Shin , “RB-Seeker: auto-detection
of redirection Botnets” in Proceedings of 16th Annual
Network & Distributed System Security Symposium (NDSS
’09), 2009.
[12] J. Grizzard , V. Sharma, C. Nunnery, B. Kang , and D.
Dagon , “Peer-to-peer botnets: Overview and case study” in
USENIX Workshop on Hot Topics in Understanding
Botnets (HotBots’07), 2007.
[13] G. Gu , R. Perdisci , J. Zhang and W. Lee , “Bot-Miner:
Clustering Analysis of Network Traffic for Protocol- and
Structure-Independent Botnet Detection” in USENIX
Security Symposium, 2008.
[14] J. Govil, “Examining the criminology of bot zoo,” in
Proceedings of the 6th International Conference on
Information, Communications and Signal Processing
(ICICS ’07), pp. 1–6, Singapore, December 2007.
[15] P. Bacher, T. Holz, M. Kotter, and G. Wicherski, “Know
your Enemy: Tracking Botnets,”
http://www.honeynet.org/papers/ bots., Accessed:
September 2009
[16] K. Pappas, “Back to basics to fight botnets,”
Communications News, vol. 45, no. 5, p. 12, 2008.
[17] P. Sroufe, S. Phithakkitnukoon, R. Dantu, and J. Cangussu,
“Email shape analysis for spam botnet detection,” in
Proceedings of the 6th IEEE Consumer Communications
and Networking Conference (CCNC ’09), pp. 1–2, Las
Vegas, Nev, USA, January 2009.
[18] Info World Newsletter.
http://www.infoworld.com/d/security-central/botnets-new-
political-activism-392 Accessed November, 2009
[19] P. Ferguson, D. Senie, “Network ingress filtering:
Defeating Denial of Service attacks which employ IP
 source address spoofing,” RFC 2267, the Internet
Engineering Task Force (IETF), 1998
[20] V. Paxson. BRO: A System for Detecting Network
Intruders in Real Time. In Proceedings of the 7th USENIX
Security Symposium, 1998.
[21] M. Roesch. Snort - lightweight intrusion detection for
networks. In Proceedings of USENIX LISA’99, 1999,
Accessed November, 2009
[22] Mandujano, S., Galván, A. Ountbound Intrusion Detection,
Center for Intelligent System Monterrey, Mexico, 2004
[23] J. Goebel and T. Holz. Rishi: Identify bot contaminated
hosts by irc nickname evaluation. In USENIX Workshop on
Hot Topics in Understanding Botnets (HotBots’07), 2007.
[24] W. T. Strayer, R.Walsh, C. Livadas, and D. Lapsley,
“Detecting botnets with tight command and control,” in
Proceedings of the 31st Annual IEEE Conference on Local
Computer Networks (LCN ’06), pp. 195–202, Tampa, Fla,
USA, November 2006.
[25] W. Cui, R. H. Katz, W. Tan. Design and Implementation of
an Extrusion-based Break-In Detector for Personal
Computers in Annual Computer Security Applications
Conf., Dec. 2005.
[26] E. Stinson, J. C. Mitchell. Characterizing Bots’ Remote
Control Behaviour in Detection of Intrusions & Malware,
and Vulnerability Assessment, July 2007.