You are on page 1of 6


discussions, stats, and author profiles for this publication at:

An Internet traffic analysis method with


Conference Paper May 2010

DOI: 10.1109/NOMSW.2010.5486551 Source: IEEE Xplore


36 742

3 authors, including:

Youngseok Lee Wonchul Kang

Chungnam National University Chungnam National University


Available from: Youngseok Lee

Retrieved on: 29 September 2016
An Internet Trafc Analysis Method with
Youngseok Lee Wonchul Kang Hyeongu Son
Chungnam National University Chungnam National University Chungnam National University
Daejeon, 305-764, Republic of Korea Daejeon, 305-764, Republic of Korea Daejeon, 305-764, Republic of Korea

AbstractInternet trafc measurement and analysis have been data. Typically, ISPs employ a high-performance server with
usually performed on a high performance server that collects a large storage system to collect and analyze ow data from
and examines packet or ow traces. However, when we monitor many routers. However, when we anatomize trafc in a large-
a large volume of trafc data for detailed statistics, a long-
period or a large-scale network, it is not easy to handle Tera scale network, we are often confronted with hard challenges
or Peta-byte trafc data with a single server. Common ways to of handling a huge amount of trafc data for processing and
reduce a large volume of continuously monitored trafc data are management. For example, when ISPs monitor trafc in a
packet sampling or ow aggregation that results in coarse trafc nation-wide network consisting of hundreds or thousands of
statistics. As distributed parallel processing schemes have been routers capable of exporting Cisco NetFlow data, it is not
recently developed due to the cloud computing platform and the
cluster lesystem, they could be usefully applied to analyzing easy to compute trafc statistics from many large ow les
big trafc data. Thus, in this paper, we propose an Internet ow in short time. In order to lessen the volume of continuously
analysis method based on the MapReduce software framework of streaming ow data, we normally use packet sampling or ow
the cloud computing platform for a large-scale network. From the aggregation techniques. Otherwise, after processing packet
experiments with an open-source MapReduce system, Hadoop, traces, we leave only the statistics information results.
we have veried that the MapReduce-based ow analysis method
improves the ow statistics computation time by 72%, when When we analyze ow data for a large-scale network, we
compared with the popular ow data processing tool, ow-tools, need to handle and manage a few Tera or Peta-byte packet
on a single host. In addition, we showed that MapReduce-based or ow les simultaneously. When the outbreak of global
programs complete the ow analysis job against a single node Internet worms or DDoS attacks happens, we also have to
failure. process fast a large volume of ow data at once. Yet, with a
Keywords: MapReduce, Hadoop, cloud computing, NetFlow,
trafc monitoring single-server approach, we are not able to cope efciently and
quickly with a large measurement data for scalable storage
I. I NTRODUCTION and analysis. From recent developments of cluster lesystems
In Internet trafc measurement and analysis, ow-based and cloud computing platforms, we could benet two features:
trafc monitoring methods are widely deployed throughout distributed parallel computing and fault tolerance. Google,
Internet Service Providers (ISPs), because the volume of Yahoo, Amazon, and Facebook are rigorously developing
processed data is reduced and many convenient ow statistics or making use of cluster lesystems and cloud computing
tools are available. Cisco NetFlow [1] is the popular ow platforms. Google has rst developed the MapReduce [8]
monitoring format with which we could easily monitor ows programming model for page ranking or web log analysis.
passing through routers or switches without observing every MapReduce is a software framework that supports distributed
packet. Though routers or switches do not support NetFlow, computing with two functions of map and reduce on large
we could use NetFlow-compatible ow generators such as data sets on clusters. Google operates thousands of machines
nProbe [2] to monitor packet streams in ow units. Based for MapReduce to process large web data sets. After Google
on Cisco NetFlow v9, IETF standardizes the ow-based trafc announced the MapReduce model, Yahoo has released an
monitoring method at the IP Flow Information eXport (IPFIX) open-source system for the cloud computing platform, called
[3] working group. As networks grow and evolve, we have Hadoop [9], which could process easily very large les with
to manage and monitor more and more switches and routers streaming access patterns. Amazon provides the Hadoop-based
for security, trafc engineering, Quality of Service (QoS), and cloud computing service such as Elastic Compute Cloud (EC2)
accounting reasons. or Simple Storage Service (S3). Facebook also uses Hadoop
Generally, Internet trafc measurement and analysis are to analyze the web log data for its social network service.
executed on a high performance central server. Popular tools On the other hand, cloud computing on cluster lesystems
such as tcpdump[4] or Coralreef [5] are usually run on a single provides easy fault-tolerant services for managing a huge
host to capture and process packets at a specic monitoring amount of large les. Moreover, we could build the Hadoop-
point. Flow analysis tools such as ow-tools [6] or owscan based large ow analysis system inexpensively with the com-
[7] are widely used to generate trafc statistics with NetFlow modity hardware. Hence, it is useful to apply distributed

978-1-4244-6039-7/10/$26.00 2010
c IEEE 357
parallel computing environments of cluster lesystems and Network B
cloud computing systems to the large-scale Internet trafc
Network A Network C
measurement and analysis application. Router
In this paper, we propose an Internet ow analysis method
on the cloud computing platform. Specically, we present
a MapReduce-based ow analysis scheme that could easily
process Tera or Peta-byte ow les collected from many
routers or monitoring servers. From experiments on our testbed Cluster Node
with four Hadoop data nodes, we achieved that ow statistics
computation time for large ow les could dramatically de-
crease when compared with a popular ow analysis tool run
on a single host. In addition, we showed that the MapReduce- Cloud Cluster Master
based ow analysis program nishes successfully against a Platform
single-machine failure.
The remainder of this paper is organized as follows. In Fig. 1. Architecture of the proposed ow measurement and analysis system
Section 2, we describe the related work. Our MapReduce-
based ow analysis method is explained in Section 3, and its save and process ow data as well as to manage the cluster
experimental results are presented in Section 4. Finally Section conguration. When ow data are archived on the cluster
5 concludes this paper. lesystem, the MapReduce ow analysis program is run on
the cloud platform.
Flow analysis tools such as ow-tools, owscan or Coral-
Reef are popular and widely used for generating ow statistics
such as port or protocol breakdown, because the port-based Cluster File Mapper Reducer
trafc classication method is reasonable for well-known Flow (Hadoop
Internet applications as shown in [10]. These ow monitoring collector Distributed MapReduce Library
tools are usually run on a single server with a large storage (flow- Filesystem: HDFS)
system such as RAID or Network Attached Storage (NAS). To
reduce a large volume of observed ow data, these tools often Java Virtual Machine
aggregate ows or save only ow statistics after analyzing ow
les. Yet, with these tools, it is not easy to process quickly Operating System ( Linux x86 )
a few Tera or Peta-byte ow data in a parallel or distributed
Hardware ( HDD, Memory, NIC )
For analyzing trafc by parallel processing, several solutions
have been proposed. Among them, DIPStorage [11] uses a P2P
platform, called storage tanks, to process ow data in a parallel
Fig. 2. Functional components of a cluster node
way. However, each storage tank associated with a specic
ow processing rule might be increasing the computation Each cluster node is equipped with a ow collector, a
overhead. distributed cluster lesystem, and a MapReduce library as
Recently many MapReduce programs have been developed shown in Fig. 2. A ow collector receives ow packets,
by Google, Yahoo, Amazon, etc. to analyze big data of web stores them to les, and moves ow les at local disk to the
search documents, text les or web log. Chen et al. developed cluster lesystem. NetFlow packets from routers or monitoring
a snort [12] log data analysis method with Hadoop [13] for a servers are usually sent to cluster nodes in unicast. Since
large-scale network security application. Hive [14] and HBase NetFlow packets are delivered in UDP, it does not guarantee
[15] are often used with Hadoop for easy management of big the reliability. In IPFIX, we could use SCTP instead of UDP
data. To the best of our knowledge, however, our work is the for reliability. Anycasting could be used like DNS to provide
rst study to propose an Internet-scale ow analysis method load balancing with the cluster nodes when receiving NetFlow
with MapReduce. packets. Then, ow data are saved into les associated with
each ow-exporting router periodically (e.g., ve minutes),
which will be uploaded to the cluster lesystem. For the ow
A. Overview collector, we use ow-tools for the NetFlow collecting and
Figure 1 shows the architecture of our ow measurement processing tool. Mapper and Reducer will analyze ow data
and analysis system. The cloud platform provides the cluster with Hadoop MapReduce library. To meet the customized
lesystem and the cloud computing functions. Flow data from purpose of ow analysis, we could implement appropriate
routers are delivered to the cluster through unicasting or Mapper and Reducer programs. The distributed lesystem
anycasting. Cluster nodes are operated by a master node to provides easy management of very large les as well as fault-

358 2010 IEEE/IFIP Network Operations and Management Symposium Workshops


Duration Flow count Flow le count Total binary le size Total text le size
(million) (GB) (GB)
1 day 3.2 228 0.2 1.2
1 week 19.0 1596 0.3 2.3
1 month 109.1 7068 2.0 13.1

tolerant service. For our cloud computing platform, we employ well-known Internet trafc statistics programs. Figure 3 shows
Hadoop that provides open MapReduce software framework the procedure of the MapReduce-based program that performs
and cluster le system on the Java virtual machine (VM). port-breakdown analysis of ow data.
HDFS is suitable for handling very large les with the stream- 1) Input ow les: At rst, after storing ow data from
ing data access pattern that is a write-once and read-many- ow probes on the local disk, we move the raw NetFlow
times pattern. In HDFS, a name node operates management of v5 les to the cluster lesystem, HDFS. As the current
the lesystem metadata and provides management and control Hadoop mapper supports only text les for the input
services, while a data node supplies block storage and retrieval format, we convert NetFlow les to text-format ones.
services. A name node at the master will perform recovery and As the size of text-format ow les is much larger than
automatic backup of name nodes. The block size (64 MB by that of binary-format ones, we need to support binary
default) and the number of replicated blocks in HDFS could ow les to the inputs for Mapper. Otherwise, the gzip-
be recongured according to the fault-tolerance policy. compressed text ow les could be used for the input
B. Flow Analysis Method with MapReduce format.
2) Mapper: Our ow mapper reads each ow record split
In the MapReduce programming model, the computation by new lines. A ow record has attributes of timestamp,
takes a set of input key/value pairs, and produces a set of out- IP addresses, port, protocol, ag, octet count, packet
put key/value pairs. Map and Reduce are two basic functions count, and interface numbers. Though we use only the
in the MapReduce computation. Users write Map that takes NetFlow v5 ow format in this work, we could extend
an input pair and produces intermediate key/value pairs. The the supported ow format to NetFlow v9 or IPFIX.
Hadoop MapReduce library will group the intermediate values After reading a ow record, we lter out necessary ow
according to the same key. Reduce that is also written by users attributes for a ow analysis job. As shown in Fig. 3,
will merge the intermediate values for smaller values. when the ow analysis job sums up octet counts per
destination port number, we set key/value pairs as (dst
port, octets). The ow map task will write its temporary
Flow File results on the local disk.
Start time, End time, Sif, SrcIPaddr, SrcPort, Dif, DstIPaddr, DstPort, Proto, Flag, Pkts, Octets 3) Reducer: The ow reducer will be called with the
inputs as the intermediate values generated by ow
Mapper Temp Reducer
mappers. As in the port-breakdown example, a value
(DstPort, Octets) list of octets belonging to the same destination port
------------------ Read Temp Data
(80, 1000) (DstPort, list(Octects)) number will be summed up. After merging octet values
Read Line (53, 128) (80, [1000, 200, 500]) associated with the destination port, the ow reducer
(80, 200) (53, [128, 64])
(80, 500)
writes the octet value for each port number.
(53, 64)
Parsing DstPort & Sum & Make
. (80, 1700)
. (53, 192]) A. Experimental environment
Add Result Set
For the performance evaluation of ow analysis with
Add to File MapReduce, we built up a small Hadoop testbed consisting
of a master node and four data nodes. Each node has quad-
core 2.83 GHz CPU, 4 GB memory, and 1.5 TB hard disk.
Fig. 3. A MapReduce ow analysis program for destination port breakdown
HDFS is used for the cluster lesystem. All Hadoop nodes are
To implement various ow analysis programs with MapRe- connected with 1 Gigabit Ethernet cards.
duce, we have to determine appropriate input key/value pairs With ow-tools we collected NetFlow v5 packets sent by
for each analysis program. For example, when we analyze a ow generation tool, nProbe, that exports ow data for a
trafc by port breakdown, which sums up the octet count for Gigabit Ethernet link in our campus network. Then, we saved
the port number, the key/value pair will be (port, octet). In exported ows on a le every ve minutes. As we captured
this work, we have written a simple port-breakdown program ow data on a small network of a /24 prex subnet, a ve-
for the performance evaluation. With MapReduce, accordingly, minute ow le is not enough to assess the performance of
we could realize typical ow analysis functions provided by MapReduce. Thus, to evaluate the ow statistics computation

2010 IEEE/IFIP Network Operations and Management Symposium Workshops 359

time for large data sets, we used input ow les collected for ow analysis method against failures to the cloud computing
one day, one week, and one month in Table I. One-day ow platform. Among several failure scenarios, we have experi-
les include about 3.2 million ow records, one-week ow mented two fault recovery cases for 3.2 million ows as shown
les 19.0 million, and one-month ow les 108.1 million. The in Fig. 5 and Fig. 6 where a single Hadoop data node fails
binary ow les are used inputs to ow-tools, whereas the text when either a Map or Reduce task is in progress. First, Fig.
ow les to our MapReduce program. 5 depicts how the Map task is recovered after a node among
B. Flow statistics computation time four Hadoop data nodes running Map tasks is forced to reboot.
A Hadoop data node fails at 4 seconds when the completion
percentage of Map tasks is only 9%. Thus, the Map task is re-
Port-break Computation Time
executed at 266 seconds when it nds that the intermediately-
flow-tools mapped results are not complete even though it has already
MR(2) reached 100%. Second, we can observe that the Reduce task is
3.5 MR(4) recovered at 320 seconds after a Hadoop node running Reduce
tasks is shutdown at 29 seconds as shown in Fig. 6. When a
Time (hour)

2.5 failure occurs, it takes longer time for MapReduce to complete
2 the task as given in Table II. Under a large data set of 108.2
1.5 million ows, however, MapReduce with four data nodes spent
only 1.5 times more seconds to complete the job by recovering
Map/Reduce failures. Through experiments, we have veried
3.2 19.0 40 60 80 108.1 that the ow computation job could successfully nish against
Number of flows (million) a single node failure through the Hadoop fault-tolerant service.
Fig. 4. Destination port breakdown completion time:ow-tools vs. MapRe-
When Map Fails
We compare the popular ow statistics program, ow-
tools, on a single server with our ow MapReduce pro- Task Completion Percentage (%) 100
Recovery Begins
gram in Fig. 4. The purpose of tested programs is to
compute the octet count for each destination port number.
We ran flow-cat /flowdirectory/ | flow-stat 60
-f 5 > result commands of ow-tools to concatenate
binary ow les stored in a directory and to calculate the ow 40
statistics for the destination port. Our MapReduce program
reads text ow les and produces the octet count for each 20 Map
destination port. To observe the impacts of the number of Map Node Fails
data nodes on the performance of the MapReduce program, 0 100 200 300 400 500
we carried out the experiments with 1, 2, 3, and 4 data nodes. Time (sec)
As shown in Fig. 4, the port-breakdown computation time
Fig. 5. MapReduce failure recovery: when Map fails
of ow-tools increases linearly as the number of input ows
grows, whereas that of MapReduce does not quickly build
up. With a single Hadoop data node, MR(1), the MapReduce
program for the input les of 3.2 million and 19.0 million
ow records does not outperform ow-tools. However, when When Reduce Fails
we tested the MapReduce program with two or more Hadoop 100
Task Completion Percentage (%)

Recovery Begins
data nodes (MR(2), MR(3), and MR(4)), we obtained that
MapReduce reduces dramatically the ow computation time 80
for all input les. Given the input les of 108.1 million ow
records, it took 4.5 hours for ow-tools to complete the job, 60

whereas only 1.25 hours for MapReduce with four Hadoop

data nodes, MR(4). The ow statistics computation time of
MR(4) has decreased by 72% for 108.1 million ow records. 20 Map
From the experiments, we could verify that the MapReduce Reduce Node Fails

ow analysis method computes efciently ow statistics for a 0

0 100 200 300 400 500
big data set, which will be scalable in a large-scale network.
Time (sec)
C. Recovery of a single node failure
Fig. 6. MapReduce failure recovery: when Reduce fails
As processes or machines often fail because of software
or hardware malfunctions, we need to provide a fault-tolerant

360 2010 IEEE/IFIP Network Operations and Management Symposium Workshops

TASK COMPLETION TIME WITH OR WITHOUT A SINGLE NODE FAILURE [14] Ashish Thusoo, Joydeep Sen Sarma, Namit Jain, Zheng Shao, Prasad
Chakka, Suresh Anthony, Hao Liu, Pete Wyckoff, Raghotham Murthy
Flow counts Time of MR(4) Time of MR(4) with failure Hive: a warehousing solution over a map-reduce framework., Proceedings
(seconds) (seconds) of the VLDB Endowment Volume 2 , Issue 2 (August 2009) Pages: 1626-
3.2 M 220.2 380.2 1629
19.0 M 2096.7 2588.5 [15] HBase,
108.2 M 4549.1 6754.4

In this paper, we presented a MapReduce-based ow anal-
ysis method for a large-scale networks that could analyze
efciently and quickly big ow data against failures. On the
Hadoop system, we have evaluated the performance of the
MapReduce-based ow analysis method by developing a port-
breakdown program. From the experiments with four Hadoop
data nodes, we achieved that ow computation time could
be dramatically improved by 72% compared with the typical
ow analysis tools. In addition, we showed that the fault-
tolerant service against a single machine failure could be
easily provided by MapReduce-based ow analysis. Though
our MapReduce-based ow analysis scheme outperforms the
legacy single-host tools, we need to improve a few drawbacks
of the current MapReduce-based approach such as batch
processing jobs or text input le formats, and to develop
convenient ow analysis tools based on MapReduce.

This work was partly supported by the MKE(Ministry of
Knowledge Economy), Korea, under the ITRC(Information
Technology Research Center) support program supervised by
the NIPA(National IT Industry Promotion Agency) (NIPA-
2010-(C1090-0902-0016)), and partly by the IT R&D pro-
gram of MKE/KEIT [KI001878, CASFI(Collect, Analyze, and
Share for Future Internet): High-Precision Measurement and
Analysis Research].

[1] Cisco NetFlow,
[2] L. Deri, nProbe: an Open Source NetFlow Probe for Gigabit Networks,
TERENA Networking Conference, May 2003.
[3] J. Quittek, T. Zseby, B. Claise, and S. Zander, Requirements for IP Flow
Information Export (IPFIX), IETF RFC 3917, October 2004.
[4] tcpdump,
[5] CAIDA CoralReef Software Suite, alreef.
[6] M. Fullmer and S. Romig, The OSU Flow-tools Package and Cisco
NetFlow Logs, USENIX LISA, 2000.
[7] D. Plonka, FlowScan: a Network Trafc Flow Reporting and Visualizing
Tool, USENIX Conference on System Administration, 2000.
[8] J. Dean and S. Ghemawat, MapReduce: Simplied Data Processing on
Large Cluster, OSDI, 2004.
[9] Hadoop,
[10] H. Kim, K. Claffy, M. Fomenkov, D. Barman, M. Faloutsos, and K. Lee
Internet Trafc Classication Demystied: Myths, Caveats, and the Best
Practices, ACM CoNEXT, 2008.
[11] C. Morariu, T. Kramis, B. Stiller DIPStorage: Distributed Architecture
for Storage of IP Flow Records., 16thWorkshop on Local and Metropoli-
tan Area Networks, September 2008.
[12] M. Roesch, Snort - Lightweight Intrusion Detection for Networks,
[13] W. Chen and J. Wang, Building a Cloud Computing Analysis System for
Intrusion Detection System, CloudSlam 2009.

2010 IEEE/IFIP Network Operations and Management Symposium Workshops 361