Inside the Perimeter:

Six Steps to Improve

Your Security
Monitoring

BRKSEC-2006

BRKSEC-2006
17796_04_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 2

© 2006, Cisco Systems, Inc. All rights reserved. 1

Presentation_ID.scr
 Cisco TelePresence
Next-Generation IP Video Conferencing

BRKSEC-2006
17796_04_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 33

TelePresence Public Launch Schedule

10 – 12:30 AM
Press
8-9 am
2:30-5 PM
15 Customers 3-5:30 PM
Press
10:30-12:30 Rob Lloyd Press
Charlie Giancarlo 11:30-2:00
12 Customers* Marthin De Beer
15 Customers
Rick Justice 5:30-8 PM
John Chambers Chris Dedicoat
5:30-6:30 PM Randy Harrell Press
1:30-3:30 Sue Bostrom
(8:30-9:30 AM HK) Charles Stucki
10 Press
15 Customers
10 Press Rick Justice
John Chambers Rob Lloyd
Charleston Sin Randy Harrell Charlie Giancarlo
Owen Chan Marthin De Beer
Guido Jouret

Hong Kong San Jose NYC London

24th Oct 23rd Oct 23rd Oct 23rd Oct
BRKSEC-2006
17796_04_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 4

© 2006, Cisco Systems, Inc. All rights reserved. 2

Presentation_ID.scr
 TelePresence Monitoring Architecture
Cisco IDS, NetFlow, and CS-MARS

Data Center

NetFlow
anomaly detection CCM CCM
424 424

Cisco IDS
CSIRT monitoring CS-MARS
IDS events
signature detection

BRKSEC-2006
17796_04_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 5

False Positive Traffic Example:

SSH Sync Between CM’s

False Positive:
normal sync traffic
between call
managers

BRKSEC-2006
17796_04_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 6

© 2006, Cisco Systems, Inc. All rights reserved. 3

Presentation_ID.scr
 Security Event Example:
Infected Host Attacking Call Managers

IDS and MARS

detecting hosts
attacking call
managers

Attacking host was blackholed and submitted for remediation

BRKSEC-2006
17796_04_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 7

6. Troubleshoot
Six Steps to 5. Feed and tune
Improve Your 4. Choose event sources
Security
3. Select targets
Monitoring
2. Know the network

1. Define your policy

BRKSEC-2006
17796_04_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 88

© 2006, Cisco Systems, Inc. All rights reserved. 4

Presentation_ID.scr
 What to Expect From This Session
ƒ Practical advice
ƒ Focused on monitoring, IR
ƒ Assumes:
Experience with monitoring, IR
Experience with tools

ƒ Shares experience moving

from DMZ focus to
internal focus

BRKSEC-2006
17796_04_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 99

Scenario: Blanco Wireless

ƒ Blanco Wireless sells mobile
phone service
ƒ Gathers SSNs from customers
to activate service (stored in
database)
ƒ Running Oracle 10g app suite
ƒ Monitor to protect data and
comply with government
regulations

BRKSEC-2006
17796_04_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 10

© 2006, Cisco Systems, Inc. All rights reserved. 5

Presentation_ID.scr
 Step 1.
Build and
Understand
BRKSEC-2006
17796_04_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public
Your Policy
11

Monitor Against Defined Policies

ƒ Which policies to monitor?
Be concrete, precise
Which will management
enforce?
ƒ Types of policies
Compliance with regulations or
standards
SOX – monitor financial apps and
databases
HIPAA – monitor healthcare apps
and databases
ISO 17799 - best practices for
information security
Employee policies
Rogue devices – laptops, wireless,
DC devices, honeypots, etc.
Employees using shared accounts
Hardened DMZ devices – services
running that should not be?
Direct login with privileged
accounts (root, DBA, etc.)
BRKSEC-2006 Tunneled traffic – P2P, etc. 12
17796_04_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public

© 2006, Cisco Systems, Inc. All rights reserved. 6

Presentation_ID.scr
 Example: COBIT DS9.4,
Configuration Control
Monitor Changes to Network Devices, Reconcile
Against Approved Change Lists

Who changed
the Pix config?

BRKSEC-2006
17796_04_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 13

More Policy Monitoring Examples

ƒ Policy: No direct privileged logins

Monitor IDS, SSH logs for successful root logins

ƒ Policy: Use strong passwords

Vulnerability scan for routers with Cisco/Cisco credentials

ƒ Policy: No internet access from production servers

Monitor for accepted connections to Internet initiated
from servers

ƒ Policy: No protocol tunneling

Monitor IDS alerts for protocols tunneled over DNS to/from
non-DNS servers

BRKSEC-2006
17796_04_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 14

© 2006, Cisco Systems, Inc. All rights reserved. 7

Presentation_ID.scr
 Example: FTP Root Login
evIdsAlert: eventId="1173129985693574851" severity="low" vendor="Cisco"
originator:
hostId: rcdn4-dmz-nms-1
appName: sensorApp
appInstanceId: 421
time: Mar 22 2007 18:14:39 EDT (1174601679880242000) offset="0" timeZone="UTC"
signature: version="S31" description="Ftp Priviledged Login" id="3171"
subsigId: 1
sigDetails: USER administrator
marsCategory: Info/SuccessfulLogin/FTP
interfaceGroup: vs0
vlan: 0 Caught
participants:
attacker: successful FTP
addr: 163.180.17.91 locality="OUT"
port: 1387 Administrator
target:
addr: 12.19.88.226 locality="IN"
login via IDS
port: 21
os: idSource="unknown" relevance="unknown" type="unknown"
summary: 2 final="true" initialAlert="1173129985693574773" summaryType="Regular"

alertDetails: Regular Summary: 2 events this interval ;

riskRatingValue: 37 targetValueRating="medium"
threatRatingValue: 37
interface: ge0_0
protocol: tcp
BRKSEC-2006
17796_04_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 15

Example: SSH root login message

Mar 28 16:19:01 xianshield sshd(pam_unix)[13698]:

session opened for user root by (uid=0)

Caught direct
root login via
syslog

BRKSEC-2006
17796_04_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 16

© 2006, Cisco Systems, Inc. All rights reserved. 8

Presentation_ID.scr
 Blanco Wireless: Policies
ƒ SSN’s must be encrypted in storage (SB1386)
ƒ Forbid copying data from production database to desktops
ƒ Database cannot initiate connections outside data center
ƒ No direct privileged logins to server or database
ƒ Database must be hardened to the DISA Database
STIG standard
No development in production
No developer logins in production

ƒ Linux servers must be hardened to RedHat’s recommended

hardening
No development in production
No developer logins in production
BRKSEC-2006
17796_04_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 17

BRKSEC-2006
17796_04_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public
Step 2: Know Your Network
18

© 2006, Cisco Systems, Inc. All rights reserved. 9

Presentation_ID.scr
 Do You Have a Self Defeating Network?

ƒ Unknown
ƒ Unmonitored
ƒ Uncontrolled
ƒ Unmanned
ƒ Trusted

Source: Richard Beijtlich

BRKSEC-2006
17796_04_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 19

What Is Meant by ‘Telemetry’?

Te·lem·e·try— a technology that allows the remote

measurement and reporting of information of interest to
the system designer or operator. The word is derived from
Greek roots tele = remote, and metron = measure

BRKSEC-2006
17796_04_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 20

© 2006, Cisco Systems, Inc. All rights reserved. 10

Presentation_ID.scr
 Network Telemetry—
What’s It Do for Me?
ƒ Historically used for
capacity planning
ƒ Detects attacks
With analysis tools, can
detect anomalies

ƒ Supports investigations
Tools can collect, trend,
and correlate activity

ƒ Well supported
Arbor PeakFlow
CS-MARS
NetQoS
OSU FlowTools

ƒ Simple to understand
BRKSEC-2006
17796_04_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 21

Network Telemetry—
Time Synchronization
ƒ Without it, can’t correlate
different sources
ƒ Enable Network Time
Protocol (NTP)
everywhere
supported by routers,
switches, firewalls, hosts,
and other network-
attached devices

ƒ Use UTC for time zones

BRKSEC-2006
17796_04_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 22

© 2006, Cisco Systems, Inc. All rights reserved. 11

Presentation_ID.scr
 What’s NetFlow?
ƒ NetFlow is a form of telemetry pushed from the
network devices
ƒ Netflow is best used in combination with other
technologies: IPS, vulnerability scanners, and full
traffic capture
Traffic capture is like a wiretap
NetFlow is like a phone bill
ƒ We can learn a lot from studying the network phone bill!
Who’s talking to whom? And when?
Over what protocols and ports?
How much data was transferred?
At what speed?
For what duration?
BRKSEC-2006
17796_04_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 23

Elements of a NetFlow Packet

Ingress i/f
NetFlow is our
Data Flow Data Flow #1 Tool
Ne
tflo

Egress i/f
w

Usage
ƒ Packet Count ƒ Source IP Address From/To
ƒ Byte Count ƒ Destination IP Address

Time ƒ Start sysUpTime ƒ Source TCP/UDP Port

Application
of Day ƒ End sysUpTime ƒ Destination TCP/UDP Port

Port ƒ Input ifIndex ƒ Next Hop Address

Utilization ƒ Output ifIndex Routing
ƒ Source AS Number and
ƒ Dest. AS Number Peering
ƒ Type of Service ƒ Source Prefix Mask
QoS
ƒ TCP Flags ƒ Dest. Prefix Mask
ƒ Protocol
BRKSEC-2006
17796_04_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 24

© 2006, Cisco Systems, Inc. All rights reserved. 12

Presentation_ID.scr
 NetFlow Setup

ƒ Don’t have a copy of NetFlow data b/c IT won’t share?

Many products have the ability to copy flow data off to other
destinations

Export netflow
data to OSU
Regionalized Flowtools
collection to Collector
minimize
WAN impact

Storage
Collector

NetFlow data
copied to other
destinations with Peakflow
NetQoS
flow-fanout

BRKSEC-2006
17796_04_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 25

NetFlow Collection at Cisco

ƒ DMZ NetFlow Collection (4 servers)
ƒ Data Center NetFlow Collection (20+ servers)
ƒ Query/Reporting tools (OSU Flowtools, DFlow, NetFlow 200K pps
Report Generator) 3 ISP gateways
600GB ~ 3 months

BRKSEC-2006
17796_04_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 26

© 2006, Cisco Systems, Inc. All rights reserved. 13

Presentation_ID.scr
 OSU Flowtools
NetFlow Collector Setup

ƒ Tool: OSU FlowTools

Free!
Developed by Ohio State
University

ƒ Examples of capabilities
Did 192.168.15.40 talk to
216.213.22.14?
What hosts and ports did
192.168.15.40 talk to?
Who’s connecting to port
TCP/6667?
Did anyone transfer data >
500MB to an external host?

BRKSEC-2006
17796_04_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 27

OSU Flowtools Example

Who’s Talking?

ƒ Scenario: New botnet, variant undetected

You need to identify all systems that ‘talked’ to the botnet C&C
Luckily you’ve deployed NetFlow collection at all your PoPs

flow.acl file uses

familiar ACL
syntax. create a
[mynfchost]$ head flow.acl put
put inin specific
specific list named ‘bot’
ip access-list standard bot permit host 69.50.180.3 concatenate all files
query
query syntax
syntax for
for
from Feb 12,
ip access-list standard bot permit host 66.182.153.176 2007
the
the example
example
then filter for src or
dest of ‘bot’
[mynfchost]$ flow-cat /var/local/flows/data/2007-02-12/ft* acl
| flow-filter -Sbot -o
-... we’ve got a
host in the
Start End Sif SrcIPaddress SrcP DIf DstIPaddress
botnet!
DstP
0213.08:39:49.911 0213.08:40:34.519 58 10.10.71.100 8343 98 69.50.180.3
31337
0213.08:40:33.590 0213.08:40:42.294 98 69.50.180.3 31337 58 10.10.71.100
83
BRKSEC-2006
17796_04_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 28

© 2006, Cisco Systems, Inc. All rights reserved. 14

Presentation_ID.scr
 Custom NetFlow Report Generator
Query by IP

BRKSEC-2006
17796_04_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 29

Know Thy Subnets

ƒ Critical to providing context to an incident
Is the address in your DMZ? lab? remote access? desktop?
data center?
ƒ Make the data queryable
Commercial and open source products available
ƒ Build the data into your security devices
SIMS—netForensics asset groups
SIMS—CS-MARS network groups
IDS—Cisco network locale variables
variables DC_NETWORKS address 10.2.121.0-10.2.121.255,10.3.120.0-10.3.127.
255,10.4.8.0-10.4.15.255
variables DMZ_PROD_NETWORKS address 198.133.219.0-198.133.219.255 Data
variables DMZ_LAB_NETWORKS 172.16.10.0-172.16.11.255
center
host!
eventId=1168468372254753459 eventType=evIdsAlert hostId=xxx-dc-nms-4appName=sensorApp
appInstanceId=6718 tmTime=1178426525155 severity=1 vLan=700 Interface=ge2_1 Protocol=tcp
riskRatingValue=26 sigId=11245 sigDetails=NICK...USER" src=10.2.121.10 srcDir=DC_NETWORKS
srcport=40266 dst=208.71.169.36 dstDir=OUT
dstport=6665
BRKSEC-2006
17796_04_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 30

© 2006, Cisco Systems, Inc. All rights reserved. 15

Presentation_ID.scr
 Network Telemetry—MRTG/RRDTool
ƒ Not just netflow, can also use SNMP to grab telemetry
ƒ Shows data volumes between endpoints You must
understand
your network
traffic volume!

BRKSEC-2006
17796_04_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 31

Blanco Wireless: Network

ƒ Network traffic data Based on our design,

environment, and these
aggregate traffic levels with
spikes above 400Mbps, We
need an IPS 4260

ƒ Subnet information - IP address management data

10.10.0.0/19 A (Active) Data Centers
|-- 10.10.0.0/20 A (Active) Building 3 Data Center
| |-- 10.10.0.0/25 S (Active) Windows Server Subnet
| |-- 10.10.0.128/25 S (Active) Oracle 10g Subnet
| |-- 10.10.1.0/26 S (Active) ESX VMWare Farm
| |-- 10.10.1.64./26 S (Active) Web Application Servers

10.10.0.0/16 A (Active) Indiana Campus

|-- 10.10.0.0/19 A (Active) Data Centers
|-- 10.10.32.0/19 A (Active) Site 1 Desktop Networks
| |-- 10.10.32.0/24 S (Active) Building 1 1st floor
| |-- 10.10.33.0/25 S (Active) Building 1 2nd floor
| |-- 10.10.33.128/25 S (Active) Building 2
BRKSEC-2006
17796_04_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 32
32

© 2006, Cisco Systems, Inc. All rights reserved. 16

Presentation_ID.scr
 BRKSEC-2006
17796_04_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved.
Step 3. Select Your Targets
Cisco Public 33

1. Determine Which Assets to Monitor

ƒ Face it: you can’t monitor
everything equally
ƒ How to prioritize?
Revenue impact?
Regulatory compliance/legal obligation?
Expense reduction?
At risk?
Systems that can’t be patched
Most attractive targets to hackers?
Sensitive data?
Visibility to upper management?
Manageable event rates?

ƒ Hopefully, someone else figured

this out for you
Disaster planning teams

ƒ Which incidents can be mitigated?

BRKSEC-2006
17796_04_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 34

© 2006, Cisco Systems, Inc. All rights reserved. 17

Presentation_ID.scr
 Recommendation: Best
Monitoring Targets
1. Access sensitive data
Legal compliance
Intellectual property
Customer sensitive data
2. Risky
Fewer controls (ACL’s, poor
configs, etc.)
Hard to patch (limited patch
windows, high uptime
requirements, custom vendor
code, etc.)
3. Generate revenue
4. Produce actionable events
Why monitor if you can’t
BRKSEC-2006
mitigate?
17796_04_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 35

2. Determine Components to Monitor

ƒ What assets are associated with the target?
Host names
Databases
Applications
Network devices
ƒ Example: Monitor ERP system
List assets associated with system
Ten clustered Linux servers
Five clustered database servers
Four “logical” application names
One LDAP server
Policy: Database should only be accessed
from app server
Monitor for:
Outbound connections from db
Access to DB on non SQL ports (SSH, terminal
services, etc.)
BRKSEC-2006
17796_04_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 36

© 2006, Cisco Systems, Inc. All rights reserved. 18

Presentation_ID.scr
 Blanco Wireless: Monitoring Targets
ƒ Application that
processes SSNs
Account Management
application
ƒ Components
to monitor
Web/app server
DB server
ƒ Information to gather
IP addresses of web
and db server
User names running
services
User name used to
connect to DB
DB instance name
DB schema name
Access controls in place
BRKSEC-2006
17796_04_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 37

BRKSEC-2006
17796_04_2008_c1
Step 4. Choose Event Sources
© 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 38

© 2006, Cisco Systems, Inc. All rights reserved. 19

Presentation_ID.scr
 Choosing Event Sources: What to Consider

ƒ How will you use it?

For monitoring
For incident response
For investigations
ƒ How will you collect it?
Pushed from device (syslog,
netflow, etc.)
Pulled from device (SDEE,
SNMP, Windows logs, etc.)
Detected with special
equipment (IDS, etc.)
ƒ Performance: what will it do
to the sending device?
Can you get sufficient detail?
Will the support staff give it
to you?
BRKSEC-2006
17796_04_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 39

Choosing Event Sources:

What to Consider (Cont.)
ƒ How much storage do
you have?
ƒ What tools will you use to
read it?
SIM, log analyzer, etc.
ƒ Application specific
Can you recognize “false
positive” patterns and tune
them out?
Will you get enough
information to act on it without
a full packet-capture?
Can you identify specific
incidents and how you’d see it
with your event source?
Do you know what you’d do
with it if there’s really an
incident?

BRKSEC-2006
17796_04_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 40

© 2006, Cisco Systems, Inc. All rights reserved. 20

Presentation_ID.scr
 Three Best Event Sources
ƒ NetFlow
Collect at chokepoints (data center gateways)
Cheap to collect: SJC stores three ISP gateways, 200k pps, 600GB
storage, can query back three months
Free tools to collect, relay, query
OSU FlowTools, nfdump/nfsen, etc.
ƒ Network IDS
Collect at chokepoints (data center gateways)
No agents or feeds taxing end systems
ƒ Host logs
Unix: syslog
Collect common services via syslog (web servers, mail servers, etc.)
Collect with syslog relay/collector
syslog-ng, splunk, etc.
Collect Windows logs into same infra with Snare agents
BRKSEC-2006
17796_04_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 41

Logging Has Performance Impact

ƒ Router Access
Control List (ACL) logs
High event volumes
Can impact performance
Rate-limiting possible

ƒ Pix and FWSM logs

Also generate ACL logs
Performance impact
much less than
on routers

ƒ OS and Application Logs

Syslog, messages,
authlog, access_log, etc.
BRKSEC-2006
17796_04_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 42

© 2006, Cisco Systems, Inc. All rights reserved. 21

Presentation_ID.scr
 Searching Through Logs w/Splunk

BRKSEC-2006
17796_04_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 43

Searching Through Logs w/Sawmill

BRKSEC-2006
17796_04_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 44

© 2006, Cisco Systems, Inc. All rights reserved. 22

Presentation_ID.scr
 Blanco Wireless: Event Sources
ƒ NetFlow
Connections from database out of DC
Large volume copies from DB to
desktop nets
ƒ Syslog
Direct privileged logins to Unix servers
Database, web server, and SSH server
stops, restarts
ƒ IDS
Known attacks against Oracle suite
Custom signature to watch for:
SSN pattern (###-##-####) -
unencrypted transmissions
Describe statements in DB
ƒ Oracle auditing
Queries against SSN column in
account table
Queries against V$ tables
Direct privileged logins
BRKSEC-2006
17796_04_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 45

BRKSEC-2006
17796_04_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public
Step 5. Feed and Tune
46

© 2006, Cisco Systems, Inc. All rights reserved. 23

Presentation_ID.scr
 IDS/IPS Refresher

ƒ IDS—Intrusion Detection System Evil packets

passive network traffic monitoring
limited actions, mostly for alerting

Traffic Flow

ƒ IPS—Intrusion Prevention System Match

?
inline network traffic monitoring
alerting + ability to drop packets

Alert

Traffic Flow

BRKSEC-2006
17796_04_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 47

IDS—Basic Deployment Steps

Analyze

Design

Tuning and
Deploy management
are ongoing

Tune

Manage

BRKSEC-2006
17796_04_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 48

© 2006, Cisco Systems, Inc. All rights reserved. 24

Presentation_ID.scr
 Enterprise Datacenter IPS/IDS
IDS or IPS?
ƒ Can we go inline? If so, where?
ƒ Datacenter networks typically require high availability
ƒ Your networking team has likely built in redundancy
ƒ What failover mechanisms are available in the product?
Availability = MTBF/(MTBF + MTTR)
SerialPartsAvailability = ∏(PartAvailability)
ParallelPartsAvailability = 1 - [∏(1-PartAvailability)]
CPU 1 PWR 1 Chassis

CPU 2 PWR 2
I/O Card 1 I/O Card 2

Block Diagram of a Simple Redundant System

BRKSEC-2006
17796_04_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 49

Enterprise Datacenter IPS/IDS

IDS or IPS?
Availability = MTBF/(MTBF + MTTR)
CPU 1 PWR 1 Chassis

CPU 2 PWR 2
I/O Card 1 I/O Card 2

Block Diagram of a Simple Redundant System

Chassis Availability = 400000/(400000 + 4 hrs) = 0.99999

I/O Card Availability = 200000/(200000 + 2 hrs) = 0.99999
PWR Supply Availability = 500000/(500000 + 2 hrs) = 0.999996
CPU Availability = 100000/(100000 + 2 hrs) = 0.99998
Parallel CPU Availability = 1 - [(1-0.99998) + (1-0.9998)] = 0.9999999996
Parallel PWR Supply Availability =1 - [(1-0.999996) + (1-0.999996) = 0.999999999984

System Availability = Chassis * pCPU * pPWR * Card1 * Card2 = 0.999969999884

[1 - 0.999969999884] x 525,960 Minutes/Yr = 116.76 Minutes
Source: Chris Oggerino, High Availability Network Fundamentals
BRKSEC-2006
17796_04_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 50

© 2006, Cisco Systems, Inc. All rights reserved. 25

Presentation_ID.scr
 Enterprise Datacenter IPS/IDS
IDS or IPS?
ƒ Failover/bypass mechanisms to consider
Fail open vs. fail closed MTBF for IPS Device
Power failure 80000 / (80000 + 2) = 0.999975

Redundant DC Network SNORT IPS Server

0.999994 0.999975
0.999994 * 0.999975 = 0.999969 We Just Lost Our 5th 9!!!
BRKSEC-2006
17796_04_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 51

Enterprise Datacenter IPS/IDS

ƒ Where should we go inline?
Distribution layer aggregation point could be too high of impact
Access layer VLAN is typically too much bandwidth

ƒ Target a smaller piece of the network containing critical infra

Subnet/VLAN/Segment

IDS or IPS?
IDS HERE

IPS HERE

For more details, see

BRKSEC- 3030
BRKSEC-2006
17796_04_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 52

© 2006, Cisco Systems, Inc. All rights reserved. 26

Presentation_ID.scr
 Setup IDS

ƒ Avoid asymmetry in your traffic view!

ƒ Minimize the number of platforms and designs
Small DC design, large DC design
Distribution layer router uplink traffic ideal

Ingress/
Ingress/
egress traffic only
egress traffic only

BB uplinks
mirrored to
sensor
BB uplinks
mirrored to
load
balancer

BRKSEC-2006
17796_04_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 53

Tune IDS

ƒ Use Cisco SAFE blueprint

ƒ Critical to successful deployment
ƒ Without tuning…
sensors generate alerts on all traffic
matching criteria
alerts overwhelm monitoring staff
NIDS will produce events irrelevant
to your environment
Will eventually cause NIDS to be
ignored or disabled

BRKSEC-2006
17796_04_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 54

© 2006, Cisco Systems, Inc. All rights reserved. 27

Presentation_ID.scr
 Tune IDS
ƒ Run in promiscuous mode
Default configuration
Latest signatures

ƒ Tune out benign traffic

using sensor
ƒ Tune out benign traffic w/SIM
(MARS, etc.)
Start with most frequent alerts
Trace to benign source/destination
addresses

ƒ Create location variables

Demarcations of your network (e.g.
DNS, email, DMZ)
Elucidates traffic flows
Enables more targeted investigations
Allows tailoring of filters
BRKSEC-2006
17796_04_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 55

Tune IDS Using Sensor

ƒ Tune out benign traffic using sensor

xxx-dc-ids-1# show stat virt
show statistics of the
Virtual Sensor Statistics
virtual-sensor instance
Statistics for Virtual Sensor vs0
<snip>
Per-Signature SigEvent count since reset
Sig 1101.0 = 22
Sig 3041.0 = 43
Sig 3052.0 = 1
Sig 3135.3 = 13
Sig 3159.0 = 12
Sig 5829.0 = 17 SigID 3030 is TCP
Sig 5837.0 = 22 SYN host sweep,
Sig 5847.1 = 2
need to look this one
Sig 6005.0 = 281
Sig 6054.0 = 49 up in MySDN
Sig 6055.0 = 7
Sig 3030.0 = 2045681

BRKSEC-2006
17796_04_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 56

© 2006, Cisco Systems, Inc. All rights reserved. 28

Presentation_ID.scr
 IDS Tuning
ƒ Utilize Cisco’s MySDN service for signature info—mysdn.cisco.com

Benign triggers
info tells me
that this may be
normal traffic

BRKSEC-2006
17796_04_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 57

Tune IDS Using Sensor show me all

alarms with
sigID 3030 for
the past 1
ƒ Tune out benign traffic using sensor minute
xxx-dc-nms-1# show event alert past 00:01:00 | include 3030

evIdsAlert: eventId=1173255092619515843 severity=high vendor=Cisco

originator:
hostId: xxx-dc-nms-1
appName: sensorApp
time: 2007/04/23 18:50:47 2007/04/23 18:50:47 UTC
signature: description=TCP SYN Host Sweep id=3030 version=S2
subsigId: 0 I know my network
marsCategory: Info/Misc/Scanner and this is one of our
marsCategory: Probe/FromScanner application
interfaceGroup: vs0
vlan: 0
monitoring devices, I
participants: can tune accordingly
attacker:
addr: locality=IN 10.6.30.5
target:
addr: locality=IN 10.9.4.4
port: 80

BRKSEC-2006
17796_04_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 58

© 2006, Cisco Systems, Inc. All rights reserved. 29

Presentation_ID.scr
 Tune IDS Using Sensor

ƒ Take advantage of network locale variables!

We know there are six of these mgmt systems
create a variable
xxx-dc-nms-1# conf t
xxx-dc-nms-1(config)#
called
MGT_SYSTEMS
service event-action-rules rules0
variables MGT_SYSTEMS address
create
10.6.30.5,10.6.30.6,10.30.6.7,10.50.1.5,10.50.1.6,10.50.1.7 a filter to
drop alerts for
filters insert drop_mgt_system_alerts multiple signatures
signature-id-range 4003,3030,2100,2152 when the source is
attacker-address-range $MGT_SYSTEMS the MGT_SYSTEMS
victim-address-range $IN
actions-to-remove produce-alert|produce-verbose-alert variable

ƒ If a new management server is added, I just modify the

network variable and the tuning is done!
ƒ This can be done with both hosts and networks
BRKSEC-2006
17796_04_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 59

Tune IDS Using Your SIM

ƒ Run reports of most frequent events over 24 hours

Start with the really noisy stuff
Get to where you’re not affecting data retention with high
volume false positives
On-going process: continually verify impact of new signatures

ƒ Run reports to find high severity events over 24 hours

Verify alerts have appropriate severity level for your env
Modify alert levels based on your policies and network topo
i.e. P2P coming from your datacenter should probably be
higher than the default “Informational”

BRKSEC-2006
17796_04_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 60

© 2006, Cisco Systems, Inc. All rights reserved. 30

Presentation_ID.scr
 Custom Signatures

ƒ Over 2,000 default signatures in latest pack

Covers thousands of vulnerabilities and odd traffic

ƒ Build custom signatures as needed

sig-fidelity-rating 75
sig-description
sig-name IRC BOT DOMAINS “utk.thirtyfivek.org|bla.
sig-string-info IRC BOT DOMAINS
sig-comment IRC BOT DOMAINS girlsontheblock.com”
exit
engine atomic-ip
specify-l4-protocol yes
specify-payload-inspection yes
regex-string
(\x03[Uu][Tt][Kk]\x0b[Tt][Hh][Ii][Rr][Tt][Yy][Ff][Ii][Vv][Ee][Kk]\x03[Oo][Rr][Gg
])|(\x03[Bb][Ll][Aa]\x0f[Gg][Ii][Rr][Ll][Ss][Oo][Nn][Tt][Hh][Ee][Bb][Ll][Oo][Cc]
[Kk]\x03[Cc][Oo][Mm])
exit
l4-protocol udp
specify-dst-port yes
dst-port 53

BRKSEC-2006
17796_04_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 61

Feed NetFlow to SIMs and Other Tools

ƒ Feed NetFlow to every

tool that will use it
MARS, PeakFlow, etc.

ƒ Regionalize deployment
Minimize sending over network

BRKSEC-2006
17796_04_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 62

© 2006, Cisco Systems, Inc. All rights reserved. 31

Presentation_ID.scr
 Host Syslog
ƒ Capture, store, and relay with
syslog-ng
ƒ For monitoring, be sure your
SIM can parse events
ƒ Key events to log
authentication logs
authorization logs (sudo, su, etc.)
daemon status logs (know when
they stop/start)
security application logs (tcpwrappers,
portsentry, etc.) EventID Title
528 User Logon
ƒ Windows logging Logon Failure
529 - 537
Agents can relay events via syslog 538 User Logoff
Very noisy, grab only important events 612 Audit Policy Change
517 Audit Log Cleared
BRKSEC-2006
17796_04_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 63

Other Logs
ƒ Web server logs
Can verify and elaborate attacks
Use HTTP status codes to
determine if IDS alert
really worked
Can provide URL details
during attack
Apache
Send as syslog via
httpd.conf setting
IIS
Send as syslog via
MonitorWare Agent
ƒ App server logs
SIM Find way to relay as syslog
Send via SNMP events
Pull via SQL queries
ƒ Oracle logs
Pull logs from AUD$ table via SQL
BRKSEC-2006
17796_04_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 64

© 2006, Cisco Systems, Inc. All rights reserved. 32

Presentation_ID.scr
 Perimeter vs. Internal Monitoring

What’s the Difference?

BRKSEC-2006
17796_04_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 65

Number of Services/Protocols

SMTP DNS
FTP HTTP
HTTPS SSH DMZ SMTP DNS FTP HTTP HTTPS
SSH POP IMAP NFS TFTP
TACACS TELNET NETBIOS
POP IMAP SUNRPC NNTP NTP SQL
SNMP LDAP SMB CIFS RPC

ICMP DCOM NIS DHCP TNS WINS

Datacenter ONS RADIUS HSRP VRRP

SCCP SIP H323 Q931 RTP
VNC RTSP MSEXCHANGE
iSCSI RSYNC LOTUSNOTES
RDP HTTP-ALT X11 XDCMP
ICMP SQLNET XNS SFTP IPC
MSDP LDP VERITAS PXE

ƒ Many more false positives sources

ƒ Tuning more complex
Good relationship with IT application and service owners is key

BRKSEC-2006
17796_04_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 66

© 2006, Cisco Systems, Inc. All rights reserved. 33

Presentation_ID.scr
 Enterprise Datacenter Monitoring
Complications/Difficulties

ƒ Traffic: 100+ Gbps globally vs.

4 Gbps outside
ƒ Protocols: Higher number of
services/protocols increases
variety and complexity of tuning
ƒ Alerts: Untuned sensor in large
datacenter generates > 100
million alerts/day

BRKSEC-2006
17796_04_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 67

Enterprise Datacenter Monitoring

Complications/Difficulties (Cont.)

ƒ Higher availability expectations

Enterprise data centers have very high availability requirements
Inline “IPS” a hard sell, most hardware not properly redundant
We don’t use inline IPS

ƒ False positives
Difficult and time consuming to identify
Key: good relationship with IT application and service owners

ƒ Relatively new technology

Not well understood by IDS and SIMs yet
Limited signature base
Most signatures based on Internet attacks
BRKSEC-2006
17796_04_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 68

© 2006, Cisco Systems, Inc. All rights reserved. 34

Presentation_ID.scr
 False Positives—Examples (Cisco IPS)
ƒ SMB: ADMIN$ Hidden Share Access Attempt
An attempt has been made to connect to the hidden windows
administration share ADMIN$. This share point does not appear in
normal browsing and may access attempts may be an attempt to break
into the system.
ƒ Windows RPC Race Condition Exploitation
This signature fires when detecting multiple Microsoft DCOM RPC
connection attempts in a short period of time. This may be indicative of
someone attempting to exploit the race condition that is possible.
ƒ Google Appliance ProxyStyleSheet Command Execution
This signature fires when detecting a HTTP request to a Google
appliance referring to a remote XSL stylesheet.
ƒ Multiple Rapid SSH Connections
This signature fires when there are rapid SSH connection from the
same source to the same destination
BRKSEC-2006
17796_04_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 69

Blanco Wireless: Getting NetFlow

ƒ Use a tool such as flow-dscan or nfdump to write an

alert to syslog for file transfers sourced from the
Oracle10g subnet
flow.acl file uses
familiar ACL syntax.
create a list named concatenate all files
‘oracle10g’ from May 5, 2007, filter
[mynfchost]$ head flow.acl for src of ‘oracle10g’
ip access-list standard oracle10g permit 10.10.0.128 0.0.0.127 network, write the
results to syslog

[mynfchost]$ flow-cat /var/local/flows/data/2007-05-05/ft* | flow-filter -Soracle10g |

flow-dscan -O 50000000 | logger -f outputfile -p local4

[mynfchost]# crontab -e

* * * * * su - netflow -c "env LANG=C; DATE=`date +%Y"/"%m"/"%d`; flow-cat

/var/local/flows/data/$DATE/ft* | flow-filter -Soracle10g | flow-dscan -O 500000000 |
logger -f outputfile -p local4”
add the
command to
crontab for
BRKSEC-2006 automation
17796_04_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 70
70

© 2006, Cisco Systems, Inc. All rights reserved. 35

Presentation_ID.scr
 Blanco Wireless: Using IDS

ƒ Use a custom signature to find cleartext instances of

SSN’s between any hosts except your Oracle10g web
application servers

signatures 60001 0 custom signature

! 60001
sig-description “SSN_POLICY”
sig-name SSN_POLICY
sig-string-info SSN HANDLING POLICY VIOLATION
sig-comment CLEARTEXT SSN
exit
engine string-tcp regex to match SSN
event-action produce-verbose-alert format ###-##-####
specify-min-match-length no
regex-string ([0-9][0-9][0-9]-[0-9][0-9]-[0-9][0-9][0-9][0-9])
service-ports 1-65535
exit

BRKSEC-2006
17796_04_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 71
71

Blanco Wireless: Using IDS (Cont.)

ƒ Use a custom signature to find cleartext instances of

SSN’s between any hosts except your Oracle10g web
application servers

xxx-dc-nms-1# conf t
create variables
xxx-dc-nms-1(config)#
based on IPAM data
service event-action-rules rules0
variables ORACLE_10G address 10.10.0.128-10.10.0.255
variables ORACLE_WEBAPP address 10.10.1.64-10.10.1.95
variables DESKTOP_NETS address 10.10.32.0-10.10.63.255

filters insert drop_desktop_to_webapp_alerts apply filters to

signature-id-range 60001 remove alerting
attacker-address-range $DESKTOP_NETS for the oracle
victim-address-range $ORACLE_WEBAPP web application
actions-to-remove produce-alert|produce-verbose-alert servers
filters insert drop_webapp_to_desktop_alerts
signature-id-range 60001
attacker-address-range $ORACLE_WEBAPP
victim-address-range $DESKTOP_NETS
actions-to-remove produce-alert|produce-verbose-alert

BRKSEC-2006
17796_04_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 72
72

© 2006, Cisco Systems, Inc. All rights reserved. 36

Presentation_ID.scr
 Blanco Wireless: Using IDS (Cont.)

ƒ Use a custom signature to find instances of the SQL

“describe” command against production database

signatures 60002 0 Custom signature

! 60002
sig-description “SQL_describe”
sig-name SQL_DESCRIBE
sig-string-info SQL dB enumeration
sig-comment Should not see in prod
exit regex to match
engine string-tcp SQL describe
event-action produce-verbose-alert command
specify-min-match-length no
regex-string [Dd][Ee][Ss][Cc][Rr][Ii][Bb][Ee]
service-ports 1433-`433
exit

BRKSEC-2006
17796_04_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 73
73

Blanco Wireless: Using IDS (Cont.)

ƒ Use a custom signature to find instances of the SQL

“describe” command against production database

apply filters to
remove alerting for
management
systems to dev dBs
filters insert drop_desktop_to_devdb_alerts
signature-id-range 60002
attacker-address-range $MGT_SERVERS
victim-address-range 10.10.0.120
actions-to-remove produce-alert|produce-verbose-alert

BRKSEC-2006
17796_04_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 74
74

© 2006, Cisco Systems, Inc. All rights reserved. 37

Presentation_ID.scr
 BRKSEC-2006
17796_04_2008_c1 Step 6. Maintain and Troubleshoot
© 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 75

Maintain Documented Commitments

ƒ Document agreements
with support teams
Fixed timelines
Expectations (SLAs, OS
patching, etc)
Refresh commitments
every year

ƒ Review assets regularly

Look for new assets, new
feeds, replaced hosts, etc.
Check for feeds/hosts that may
have changed/disappeared
Check for ownership changes
due to re-orgs
BRKSEC-2006
17796_04_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 76

© 2006, Cisco Systems, Inc. All rights reserved. 38

Presentation_ID.scr
 Maintain IDS Feeds
ƒ Monitor your IDS
Sensor uplinks
Sensor processes
This sensor is no ƒ Watch for
longer receiving
network traffic!
spikes/drops in
sensor alert volume
ƒ Have monitoring
staff monitor feeds

BRKSEC-2006
17796_04_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 77

Verify Feeds
ƒ Syslog feed verification
Script awk to grab
hostnames of systems
that syslog day and
do a diff
Ask IT to use a daily
cron to re-set syslog.conf
on servers

ƒ Netflow feed verification

tcpdump -i eth0 port 2060 -c
1000 | grep gw | awk '{print
$2}' | sort | uniq

BRKSEC-2006
17796_04_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 78

© 2006, Cisco Systems, Inc. All rights reserved. 39

Presentation_ID.scr
 Blanco Wireless: Maintenance
ƒ Monitor
NetFlow feed
IDS feed
Syslog feeds
Oracle auditing
ƒ Maintain agreements with:
DBA team
System administrators
Network engineers

BRKSEC-2006
17796_04_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 79

Lessons Learned
ƒ Start small
Too many events at once is
overwhelming
Understand/tune each source
before adding more
Understand “normal” traffic thoroughly
before moving on
Avoid alerting on false-positives
ƒ Use a SIM
Event correlation, false positive reduction
ƒ Choose carefully what you want
to monitor
…or you’ll waste your time chasing
false positives
ƒ Use defined playbooks, escalation
procedures
ƒ Have allies in the IT support teams
Network support, DBA’s, webmasters, etc.
BRKSEC-2006
They can explain/remediate issues you find
17796_04_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 80

© 2006, Cisco Systems, Inc. All rights reserved. 40

Presentation_ID.scr
 6. Troubleshoot
Six Steps to 5. Feed and tune
Improve Your 4. Choose event sources
Security
3. Select targets
Monitoring
2. Know the network

1. Define your policy

BRKSEC-2006
17796_04_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 81

Q and A

BRKSEC-2006
17796_04_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 82

© 2006, Cisco Systems, Inc. All rights reserved. 41

Presentation_ID.scr
 Recommended Reading

ƒ Continue your Cisco Live

learning experience with further
reading from Cisco Press
ƒ Check the Recommended
Reading flyer for suggested
books

Available Onsite at the Cisco Company Store

BRKSEC-2006
17796_04_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 83

Complete Your Online

Session Evaluation
ƒ Give us your feedback and you could win Don’t forget to activate
fabulous prizes. Winners announced daily. your Cisco Live virtual
account for access to
ƒ Receive 20 Passport points for each session all session material
evaluation you complete. on-demand and return
for our live virtual event
ƒ Complete your session evaluation online now in October 2008.
(open a browser through our wireless network Go to the Collaboration
to access our portal) or visit one of the Internet Zone in World of
stations throughout the Convention Center. Solutions or visit
www.cisco-live.com.

BRKSEC-2006
17796_04_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 84

© 2006, Cisco Systems, Inc. All rights reserved. 42

Presentation_ID.scr
 BRKSEC-2006
17796_04_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public 85

© 2006, Cisco Systems, Inc. All rights reserved. 43

Presentation_ID.scr