Professional Documents
Culture Documents
Ripe50 Plenary Tue Nfsen Nfdump PDF
Ripe50 Plenary Tue Nfsen Nfdump PDF
2005 © SWITCH
NfSen/nfdump
2005 © SWITCH 2
NfSen/nfdump
The Motivation:
NfSen and nfdump came out of operational needs.
When discussing with other teams:
– “Watch your flows for …”
– “I’ve seen a lot of … in our flows …”
But …
Router# show ip cache flow
2005 © SWITCH 3
NfSen/nfdump
Wish list:
• Must be fast!
• Must be really fast! ~ 25GB data/day
• Easy to use.
• Keep netflow data for a certain period of time.
• Easy navigation when searching stored netflow data.
• Flexible and powerful filtering.
• Flexible aggregation of netflow data.
• Top N statistics for packets, bytes, IP addresses, ports …
• Profiling hosts in case of an incident.
• A tool, which supports us in our daily work.
Many tools available, but either too slow, too cumbersome or not
what we wanted.
2005 © SWITCH 4
nfdump
nfdump:
• Stores netflow data in time sliced files.
• CMD line based tool comparable to tcpdump.
• Written in C ⇒ fast.
• Supports netflow format v5 and v7.
• Powerful pcap like filter syntax:
‘( tcp and dst net 172.16/16 and src port > 1024 and bytes < 600 ) or ( …’
• Flexible aggregation.
• Efficient filter engine: > 4 Mio flows/s on 3GHz Intel.
• Fast Statistics ( Top N ) 2.5 s for 1.5Mio flows.
Top N flows, packets, ( src/dst ) IP addresses.
• …
2005 © SWITCH 5
nfdump
List Flows:
nfsrv% nfdump -r nfcapd.200504131500 -c 10
Date flow start Len Proto Src IP Addr:Port Dst IP Addr:Port Packets Bytes
Apr 13 2005 14:59:56 0 TCP 213.161.64.210:80 -> 211.99.1.218:34156 5 828 B
Apr 13 2005 14:59:56 0 TCP 64.62.154.4:80 -> 162.139.189.158:4527 3 140 B
Apr 13 2005 14:59:56 2 TCP 131.132.112.21:1138 -> 64.18.47.234:80 5 637 B
Apr 13 2005 14:59:56 1 TCP 64.62.191.95:80 -> 172.212.81.18:4390 5 493 B
Apr 13 2005 14:59:56 0 TCP 216.109.117.206:80 -> 211.223.204.230:1132 3 266 B
Apr 13 2005 14:59:56 1 TCP 83.141.49.51:80 -> 211.92.9.56:37157 42 57.0 KB
Apr 13 2005 14:59:48 5 TCP 191.210.93.172:80 -> 149.194.8.73:3530 20 16.6 KB
Apr 13 2005 14:59:56 0 TCP 191.101.94.201:80 -> 199.53.250.100:30267 5 633 B
Apr 13 2005 14:59:56 0 TCP 199.81.104.90:60553 -> 213.161.61.209:80 6 803 B
Apr 13 2005 14:59:48 10 TCP 9.4.223.185:1433 -> 168.150.251.37:22520 3 140 B
Flows analysed: 29 matched: 10, Bytes read: 1416
Time window: Apr 13 2005 14:59:16 - Apr 13 2005 14:59:58
( IP addresses anonymised )
2005 © SWITCH 6
nfdump
( IP addresses anonymised )
2005 © SWITCH 7
nfdump
2005 © SWITCH 8
NfSen/nfdump
2005 © SWITCH 9
NfSen
Wish list:
• Use nfdump as backend tool. ⇒ modular design.
• Pictures!
• Graph current network situation.
• Graph specific profiles.
– Track hosts, ports etc. from live data.
– Profile hosts involved in incidents from history data.
• Drill down from overview to the details down to the specific flows.
• Analyse a specific time window.
• Web based.
• Automatic alerting.
• Flexible extensions using plugins.
• Easy to use.
• Auto - Cleanup. Aging data files: max space, max lifetime.
2005 © SWITCH 10
NfSen
2005 © SWITCH 11
NfSen
2005 © SWITCH 12
NfSen
( IP addresses anonymised )
2005 © SWITCH 13
NfSen/nfdump
Summary:
monitored:
...
..
Post
Web Front-end
Processing
CLI
softflowd Periodic Update Tasks & Plugins
pfflowd
Input
netflow v5, v7
nfdump Backend
2005 © SWITCH 14
NfSen/nfdump
Figures @ SWITCH:
• Server: 2 x 3GHz 2GB Ram. Debian Linux Kernel 2.6.10
• 3TB ( 2TB + 1TB ) AXUS Disk Raid
• XFS file system.
• Gigabit Ethernet interfaces.
• 5min workload avg. ca. 5%.
• 25GB Netflow data / day.
• About 41 days of netflow data available.
2005 © SWITCH 15
NfSen/nfdump in Action
2005 © SWITCH 16
NfSen/nfdump in Action
2005 © SWITCH 17
NfSen/nfdump
2005 © SWITCH 18
NfSen/nfdump
2005 © SWITCH 19
NfSen/nfdump in action
2005 © SWITCH 20
NfSen/nfdump
Profiles:
• A profile is a specific view on the netflow data with
nfdump filters applied.
• The profile applies to the graphical as well as to the
numerical view.
• Profiles can be created from data in the past. ( static )
• Profiles can be created from incoming data ( continuous )
• Any views or processing options are available.
2005 © SWITCH 21
NfSen/nfdump
Example Profiles:
Filter: ‘tcp and port 80’ Filter: ‘bytes < 100’
2005 © SWITCH 22
NfSen/nfdump
Incident Handling:
1. Customer calls and reports a hacked system:
2. Customer reports IRC connection on hacked host.
3. In agreement with the customer to find other
infected hosts ⇒ Create history profile of botnet
master.
2005 © SWITCH 23
NfSen/nfdump
Analyse Incident:
2005 © SWITCH 24
NfSen/nfdump
( IP addresses anonymised )
2005 © SWITCH 25
NfSen/nfdump
NfSen Plugins:
Post
Web Front-end
Processing
2005 © SWITCH 26
NfSen/nfdump
Plugins are:
• Simple Perl modules hooked into NfSen.
• Called at regular 5 Min intervals.
2005 © SWITCH 27
NfSen/nfdump
Plugins: #
@plugins = (
['live', 'CatchDos'],
);
1;
register
#
nfsen.conf
package CatchDos;
use strict;
#
sub Init { Report
# Init plugin …
} # End of Init …
sub run {
my $profile = shift;
my $timeslot = shift; Notification.pm
} # End of run
Runs automatically
plugin every 5 min
output
2005 © SWITCH 28
NfSen/nfdump
Example:
The plugin processes data with nfdump arguments and
filter: -A srcip,dstport -S 'bytes < 70’
Candidates for scanning activities appear:
From: nfsen@switch.ch
To: cert@switch.ch
Subject: Scanners
( IP addresses anonymised )
2005 © SWITCH 29
NfSen/nfdump
nfdump:
• Integrate Crypto - PAn:
Cryptography-based Prefix-preserving Anonymization
• Related filters: ‘Worm Footprint Tracking’
first { dst ip <A> dst port 445 bytes > 600 }
then { src ip <A> and dst ip 172.16.17.18 and dst port 80 }
• Integrate wm.edu Packeteer's PacketShaper patch into nfdump.
• Netflow v9 IPv6
• More - and more flexible statistics.
2005 © SWITCH 30
NfSen/nfdump
Summary:
• Good and flexible tools for all sort of netflow tasks.
– Network monitoring.
– Incident Handling.
– All sort of tracking …
• Open Source Tools under BSD License.
• Cmd line tool: nfdump
– Written in C. Runs on most *nix.
Tested on Linux Kernel 2.4.* and 2.6.*,
FreeBSD, OpenBSD, Solaris.
– Available at http://nfdump.sourceforge.net
• Web based frontend: NfSen
– Written in PHP and Perl.
– Extendable using plugins.
– Available at http://nfsen.sourceforge.net
• Possible candidate for the toolset in GN2/JRA2
2005 © SWITCH 31
NFSEN
2005 © SWITCH 32