Inside the Perimeter: Six Steps to Improve Your Security Monitoring

BRKSEC-2006

BRKSEC-2006 17796_04_2008_c1

© 2008 Cisco Systems, Inc. All rights reserved.

Cisco Public

2

© 2006, Cisco Systems, Inc. All rights reserved. Presentation_ID.scr

1

Cisco TelePresence
Next-Generation IP Video Conferencing

BRKSEC-2006 17796_04_2008_c1

© 2008 Cisco Systems, Inc. All rights reserved.

Cisco Public

3 3

TelePresence Public Launch Schedule
10 – 12:30 AM
Press 8-9 am

2:30-5 PM
15 Customers Press 10:30-12:30 12 Customers* Rick Justice John Chambers Randy Harrell 10 Press 10 Press Charleston Sin Owen Chan Guido Jouret 15 Customers Rick Justice John Chambers Randy Harrell Rob Lloyd Charlie Giancarlo Marthin De Beer Rob Lloyd Charlie Giancarlo Marthin De Beer

3-5:30 PM
Press 11:30-2:00

5:30-8 PM
Press 1:30-3:30

15 Customers Chris Dedicoat Sue Bostrom Charles Stucki

5:30-6:30 PM (8:30-9:30 AM HK)

Hong Kong 24th Oct
BRKSEC-2006 17796_04_2008_c1

San Jose 23rd Oct
Cisco Public

NYC 23rd Oct

London 23rd Oct
4

© 2008 Cisco Systems, Inc. All rights reserved.

© 2006, Cisco Systems, Inc. All rights reserved. Presentation_ID.scr

2

TelePresence Monitoring Architecture
Cisco IDS, NetFlow, and CS-MARS

Data Center

NetFlow anomaly detection

CCM 424

CCM 424

Cisco IDS
CSIRT monitoring CS-MARS

IDS events signature detection

BRKSEC-2006 17796_04_2008_c1

© 2008 Cisco Systems, Inc. All rights reserved.

Cisco Public

5

False Positive Traffic Example:
SSH Sync Between CM’s

False Positive: normal sync traffic between call managers

BRKSEC-2006 17796_04_2008_c1

© 2008 Cisco Systems, Inc. All rights reserved.

Cisco Public

6

© 2006, Cisco Systems, Inc. All rights reserved. Presentation_ID.scr

3

Security Event Example:
Infected Host Attacking Call Managers

IDS and MARS detecting hosts attacking call managers

Attacking host was blackholed and submitted for remediation
BRKSEC-2006 17796_04_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public

7

Six Steps to Improve Your Security Monitoring

6. Troubleshoot 5. Feed and tune
4. Choose event sources

3. Select targets 2. Know the network

1. Define your policy

BRKSEC-2006 17796_04_2008_c1

© 2008 Cisco Systems, Inc. All rights reserved.

Cisco Public

8 8

© 2006, Cisco Systems, Inc. All rights reserved. Presentation_ID.scr

4

What to Expect From This Session
Practical advice Focused on monitoring, IR Assumes:
Experience with monitoring, IR Experience with tools

Shares experience moving from DMZ focus to internal focus

BRKSEC-2006 17796_04_2008_c1

© 2008 Cisco Systems, Inc. All rights reserved.

Cisco Public

9 9

Scenario: Blanco Wireless
Blanco Wireless sells mobile phone service Gathers SSNs from customers to activate service (stored in database) Running Oracle 10g app suite Monitor to protect data and comply with government regulations

BRKSEC-2006 17796_04_2008_c1

© 2008 Cisco Systems, Inc. All rights reserved.

Cisco Public

10

© 2006, Cisco Systems, Inc. All rights reserved. Presentation_ID.scr

5

BRKSEC-2006 17796_04_2008_c1

© 2008 Cisco Systems, Inc. All rights reserved.

Cisco Public

Step 1. Build and Understand Your Policy
11

Monitor Against Defined Policies
Which policies to monitor?
Be concrete, precise Which will management enforce?

Types of policies
Compliance with regulations or standards SOX – monitor financial apps and databases HIPAA – monitor healthcare apps and databases ISO 17799 - best practices for information security Employee policies Rogue devices – laptops, wireless, DC devices, honeypots, etc. Employees using shared accounts Hardened DMZ devices – services running that should not be? Direct login with privileged accounts (root, DBA, etc.) Tunneled traffic – P2P, etc.
Cisco Public

BRKSEC-2006 17796_04_2008_c1

© 2008 Cisco Systems, Inc. All rights reserved.

12

© 2006, Cisco Systems, Inc. All rights reserved. Presentation_ID.scr

6

Example: COBIT DS9.4, Configuration Control
Monitor Changes to Network Devices, Reconcile Against Approved Change Lists

Who changed the Pix config?

BRKSEC-2006 17796_04_2008_c1

© 2008 Cisco Systems, Inc. All rights reserved.

Cisco Public

13

More Policy Monitoring Examples
Policy: No direct privileged logins
Monitor IDS, SSH logs for successful root logins

Policy: Use strong passwords
Vulnerability scan for routers with Cisco/Cisco credentials

Policy: No internet access from production servers
Monitor for accepted connections to Internet initiated from servers

Policy: No protocol tunneling
Monitor IDS alerts for protocols tunneled over DNS to/from non-DNS servers

BRKSEC-2006 17796_04_2008_c1

© 2008 Cisco Systems, Inc. All rights reserved.

Cisco Public

14

© 2006, Cisco Systems, Inc. All rights reserved. Presentation_ID.scr

7

Example: FTP Root Login
evIdsAlert: eventId="1173129985693574851" severity="low" vendor="Cisco" originator: hostId: rcdn4-dmz-nms-1 appName: sensorApp appInstanceId: 421 time: Mar 22 2007 18:14:39 EDT (1174601679880242000) offset="0" timeZone="UTC" signature: version="S31" description="Ftp Priviledged Login" id="3171" subsigId: 1 sigDetails: USER administrator marsCategory: Info/SuccessfulLogin/FTP interfaceGroup: vs0 vlan: 0 participants: attacker: addr: 163.180.17.91 locality="OUT" port: 1387 target: addr: 12.19.88.226 locality="IN" port: 21 os: idSource="unknown" relevance="unknown" type="unknown" summary: 2 final="true" initialAlert="1173129985693574773" summaryType="Regular"

Caught successful FTP Administrator login via IDS

alertDetails: Regular Summary: 2 events this interval ; riskRatingValue: 37 targetValueRating="medium" threatRatingValue: 37 interface: ge0_0 protocol: tcp
BRKSEC-2006 17796_04_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public

15

Example: SSH root login message

Mar 28 16:19:01 xianshield sshd(pam_unix)[13698]: session opened for user root by (uid=0)

Caught direct root login via syslog

BRKSEC-2006 17796_04_2008_c1

© 2008 Cisco Systems, Inc. All rights reserved.

Cisco Public

16

© 2006, Cisco Systems, Inc. All rights reserved. Presentation_ID.scr

8

Blanco Wireless: Policies
SSN’s must be encrypted in storage (SB1386) Forbid copying data from production database to desktops Database cannot initiate connections outside data center No direct privileged logins to server or database Database must be hardened to the DISA Database STIG standard
No development in production No developer logins in production

Linux servers must be hardened to RedHat’s recommended hardening
No development in production No developer logins in production
BRKSEC-2006 17796_04_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public

17

BRKSEC-2006 17796_04_2008_c1

© 2008 Cisco Systems, Inc. All rights reserved.

Cisco Public

Step 2: Know Your Network

18

© 2006, Cisco Systems, Inc. All rights reserved. Presentation_ID.scr

9

Do You Have a Self Defeating Network?
Unknown Unmonitored Uncontrolled Unmanned Trusted

Source: Richard Beijtlich
BRKSEC-2006 17796_04_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public

19

What Is Meant by ‘Telemetry’?
measurement and reporting of information of interest to the system designer or operator. The word is derived from Greek roots tele = remote, and metron = measure

Te·lem·e·try— a technology that allows the remote

BRKSEC-2006 17796_04_2008_c1

© 2008 Cisco Systems, Inc. All rights reserved.

Cisco Public

20

© 2006, Cisco Systems, Inc. All rights reserved. Presentation_ID.scr

10

Network Telemetry— What’s It Do for Me?
Historically used for capacity planning Detects attacks
With analysis tools, can detect anomalies

Supports investigations
Tools can collect, trend, and correlate activity

Well supported
Arbor PeakFlow CS-MARS NetQoS OSU FlowTools

Simple to understand
BRKSEC-2006 17796_04_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public

21

Network Telemetry— Time Synchronization
Without it, can’t correlate different sources Enable Network Time Protocol (NTP) everywhere
supported by routers, switches, firewalls, hosts, and other networkattached devices

Use UTC for time zones

BRKSEC-2006 17796_04_2008_c1

© 2008 Cisco Systems, Inc. All rights reserved.

Cisco Public

22

© 2006, Cisco Systems, Inc. All rights reserved. Presentation_ID.scr

11

What’s NetFlow?
NetFlow is a form of telemetry pushed from the network devices Netflow is best used in combination with other technologies: IPS, vulnerability scanners, and full traffic capture
Traffic capture is like a wiretap NetFlow is like a phone bill

We can learn a lot from studying the network phone bill!
Who’s talking to whom? And when? Over what protocols and ports? How much data was transferred? At what speed? For what duration?
BRKSEC-2006 17796_04_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public

23

Elements of a NetFlow Packet
Ingress i/f Data Flow
w f tflo Ne Ne
Packet Count Byte Count Start sysUpTime End sysUpTime Input ifIndex Output ifIndex Type of Service TCP Flags Protocol
Cisco Public

Data Flow

NetFlow is our #1 Tool

Egress i/f
Source IP Address Destination IP Address Source TCP/UDP Port Destination TCP/UDP Port Next Hop Address Source AS Number Dest. AS Number Source Prefix Mask Dest. Prefix Mask
From/To

Usage

Time of Day Port Utilization

Application

Routing and Peering

QoS

BRKSEC-2006 17796_04_2008_c1

© 2008 Cisco Systems, Inc. All rights reserved.

24

© 2006, Cisco Systems, Inc. All rights reserved. Presentation_ID.scr

12

NetFlow Setup
Don’t have a copy of NetFlow data b/c IT won’t share?
Many products have the ability to copy flow data off to other destinations
Export netflow data to OSU Flowtools Collector

Regionalized collection to minimize WAN impact

Storage Collector
NetFlow data copied to other destinations with flow-fanout
BRKSEC-2006 17796_04_2008_c1

Peakflow

NetQoS

© 2008 Cisco Systems, Inc. All rights reserved.

Cisco Public

25

NetFlow Collection at Cisco
DMZ NetFlow Collection (4 servers) Data Center NetFlow Collection (20+ servers) Query/Reporting tools (OSU Flowtools, DFlow, NetFlow Report Generator)
200K pps 3 ISP gateways 600GB ~ 3 months

BRKSEC-2006 17796_04_2008_c1

© 2008 Cisco Systems, Inc. All rights reserved.

Cisco Public

26

© 2006, Cisco Systems, Inc. All rights reserved. Presentation_ID.scr

13

OSU Flowtools NetFlow Collector Setup
Tool: OSU FlowTools
Free! Developed by Ohio State University

Examples of capabilities
Did 192.168.15.40 talk to 216.213.22.14? What hosts and ports did 192.168.15.40 talk to? Who’s connecting to port TCP/6667? Did anyone transfer data > 500MB to an external host?

BRKSEC-2006 17796_04_2008_c1

© 2008 Cisco Systems, Inc. All rights reserved.

Cisco Public

27

OSU Flowtools Example
Who’s Talking? Scenario: New botnet, variant undetected
You need to identify all systems that ‘talked’ to the botnet C&C Luckily you’ve deployed NetFlow collection at all your PoPs
flow.acl file uses familiar ACL syntax. create a list named ‘bot’

[mynfchost]$ head flow.acl put in specific put in specific ip access-list standard bot permit host 69.50.180.3 concatenate all files query syntax for query syntax 12, 2007 from Feb for ip access-list standard bot permit host 66.182.153.176

dest of | flow-filter -Sbot -o [mynfchost]$ flow-cat /var/local/flows/data/2007-02-12/ft* ‘bot’ acl -... we’ve got
Start DstP 0213.08:39:49.911 31337 0213.08:40:33.590 83
BRKSEC-2006 17796_04_2008_c1

then filter the example the example for src or

End

Sif

SrcIPaddress 10.10.71.100 69.50.180.3

SrcP 8343

DIf 98

a host in the DstIPaddress botnet!

0213.08:40:34.519 58 0213.08:40:42.294 98

69.50.180.3 10.10.71.100
28

31337 58

© 2008 Cisco Systems, Inc. All rights reserved.

Cisco Public

© 2006, Cisco Systems, Inc. All rights reserved. Presentation_ID.scr

14

Custom NetFlow Report Generator
Query by IP

BRKSEC-2006 17796_04_2008_c1

© 2008 Cisco Systems, Inc. All rights reserved.

Cisco Public

29

Know Thy Subnets
Critical to providing context to an incident
Is the address in your DMZ? lab? remote access? desktop? data center?

Make the data queryable
Commercial and open source products available

Build the data into your security devices
SIMS—netForensics asset groups SIMS—CS-MARS network groups IDS—Cisco network locale variables
variables DC_NETWORKS address 10.2.121.0-10.2.121.255,10.3.120.0-10.3.127. 255,10.4.8.0-10.4.15.255 variables DMZ_PROD_NETWORKS address 198.133.219.0-198.133.219.255 variables DMZ_LAB_NETWORKS 172.16.10.0-172.16.11.255

Data center host! eventId=1168468372254753459 eventType=evIdsAlert hostId=xxx-dc-nms-4appName=sensorApp
appInstanceId=6718 tmTime=1178426525155 severity=1 vLan=700 Interface=ge2_1 Protocol=tcp riskRatingValue=26 sigId=11245 sigDetails=NICK...USER" src=10.2.121.10 srcDir=DC_NETWORKS srcport=40266 dst=208.71.169.36 dstDir=OUT dstport=6665
BRKSEC-2006 17796_04_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public

30

© 2006, Cisco Systems, Inc. All rights reserved. Presentation_ID.scr

15

Network Telemetry—MRTG/RRDTool
Not just netflow, can also use SNMP to grab telemetry Shows data volumes between endpoints
You must understand your network traffic volume!

BRKSEC-2006 17796_04_2008_c1

© 2008 Cisco Systems, Inc. All rights reserved.

Cisco Public

31

Blanco Wireless: Network
Network traffic data
Based on our design, environment, and these aggregate traffic levels with spikes above 400Mbps, We need an IPS 4260

Subnet information - IP address management data
10.10.0.0/19 A (Active) Data Centers |-- 10.10.0.0/20 A (Active) Building 3 Data Center | |-- 10.10.0.0/25 S (Active) Windows Server Subnet | |-- 10.10.0.128/25 S (Active) Oracle 10g Subnet | |-- 10.10.1.0/26 S (Active) ESX VMWare Farm | |-- 10.10.1.64./26 S (Active) Web Application Servers 10.10.0.0/16 A (Active) Indiana Campus |-- 10.10.0.0/19 A (Active) Data Centers |-- 10.10.32.0/19 A (Active) Site 1 Desktop Networks | |-- 10.10.32.0/24 S (Active) Building 1 1st floor | |-- 10.10.33.0/25 S (Active) Building 1 2nd floor | |-- 10.10.33.128/25 S (Active) Building 2
BRKSEC-2006 17796_04_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public

32 32

© 2006, Cisco Systems, Inc. All rights reserved. Presentation_ID.scr

16

BRKSEC-2006 17796_04_2008_c1

© 2008 Cisco Systems, Inc. All rights reserved.

Cisco Public

Step 3. Select Your Targets
33

1. Determine Which Assets to Monitor
Face it: you can’t monitor everything equally How to prioritize?
Revenue impact? Regulatory compliance/legal obligation? Expense reduction? At risk? Systems that can’t be patched Most attractive targets to hackers? Sensitive data? Visibility to upper management? Manageable event rates?

Hopefully, someone else figured this out for you
Disaster planning teams

Which incidents can be mitigated?
BRKSEC-2006 17796_04_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public

34

© 2006, Cisco Systems, Inc. All rights reserved. Presentation_ID.scr

17

Recommendation: Best Monitoring Targets
1. Access sensitive data
Legal compliance Intellectual property Customer sensitive data

2. Risky
Fewer controls (ACL’s, poor configs, etc.) Hard to patch (limited patch windows, high uptime requirements, custom vendor code, etc.)

3. Generate revenue 4. Produce actionable events
BRKSEC-2006 17796_04_2008_c1

Why monitor if you can’t mitigate?
© 2008 Cisco Systems, Inc. All rights reserved. Cisco Public

35

2. Determine Components to Monitor
What assets are associated with the target?
Host names Databases Applications Network devices

Example: Monitor ERP system
List assets associated with system Ten clustered Linux servers Five clustered database servers Four “logical” application names One LDAP server Policy: Database should only be accessed from app server Monitor for: Outbound connections from db Access to DB on non SQL ports (SSH, terminal services, etc.)
BRKSEC-2006 17796_04_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public

36

© 2006, Cisco Systems, Inc. All rights reserved. Presentation_ID.scr

18

Blanco Wireless: Monitoring Targets
Application that processes SSNs
Account Management application

Components to monitor
Web/app server DB server

Information to gather
IP addresses of web and db server User names running services User name used to connect to DB DB instance name DB schema name Access controls in place
BRKSEC-2006 17796_04_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public

37

BRKSEC-2006 17796_04_2008_c1

© 2008 Cisco Systems, Inc. All rights reserved.

Step 4. Choose Event Sources
Cisco Public

38

© 2006, Cisco Systems, Inc. All rights reserved. Presentation_ID.scr

19

Choosing Event Sources: What to Consider
How will you use it?
For monitoring For incident response For investigations

How will you collect it?
Pushed from device (syslog, netflow, etc.) Pulled from device (SDEE, SNMP, Windows logs, etc.) Detected with special equipment (IDS, etc.)

Performance: what will it do to the sending device?
Can you get sufficient detail? Will the support staff give it to you?
BRKSEC-2006 17796_04_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public

39

Choosing Event Sources: What to Consider (Cont.)
How much storage do you have? What tools will you use to read it? SIM, log analyzer, etc. Application specific
Can you recognize “false positive” patterns and tune them out? Will you get enough information to act on it without a full packet-capture? Can you identify specific incidents and how you’d see it with your event source? Do you know what you’d do with it if there’s really an incident?
BRKSEC-2006 17796_04_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public

40

© 2006, Cisco Systems, Inc. All rights reserved. Presentation_ID.scr

20

Three Best Event Sources
NetFlow
Collect at chokepoints (data center gateways) Cheap to collect: SJC stores three ISP gateways, 200k pps, 600GB storage, can query back three months Free tools to collect, relay, query OSU FlowTools, nfdump/nfsen, etc.

Network IDS
Collect at chokepoints (data center gateways) No agents or feeds taxing end systems

Host logs
Unix: syslog Collect common services via syslog (web servers, mail servers, etc.) Collect with syslog relay/collector syslog-ng, splunk, etc. Collect Windows logs into same infra with Snare agents
BRKSEC-2006 17796_04_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public

41

Logging Has Performance Impact
Router Access Control List (ACL) logs
High event volumes Can impact performance Rate-limiting possible

Pix and FWSM logs
Also generate ACL logs Performance impact much less than on routers

OS and Application Logs
Syslog, messages, authlog, access_log, etc.
BRKSEC-2006 17796_04_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public

42

© 2006, Cisco Systems, Inc. All rights reserved. Presentation_ID.scr

21

Searching Through Logs w/Splunk

BRKSEC-2006 17796_04_2008_c1

© 2008 Cisco Systems, Inc. All rights reserved.

Cisco Public

43

Searching Through Logs w/Sawmill

BRKSEC-2006 17796_04_2008_c1

© 2008 Cisco Systems, Inc. All rights reserved.

Cisco Public

44

© 2006, Cisco Systems, Inc. All rights reserved. Presentation_ID.scr

22

Blanco Wireless: Event Sources
NetFlow
Connections from database out of DC Large volume copies from DB to desktop nets

Syslog
Direct privileged logins to Unix servers Database, web server, and SSH server stops, restarts

IDS
Known attacks against Oracle suite Custom signature to watch for: SSN pattern (###-##-####) unencrypted transmissions Describe statements in DB

Oracle auditing
Queries against SSN column in account table Queries against V$ tables Direct privileged logins
BRKSEC-2006 17796_04_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public

45

BRKSEC-2006 17796_04_2008_c1

© 2008 Cisco Systems, Inc. All rights reserved.

Cisco Public

Step 5. Feed and Tune

46

© 2006, Cisco Systems, Inc. All rights reserved. Presentation_ID.scr

23

IDS/IPS Refresher
IDS—Intrusion Detection System
passive network traffic monitoring limited actions, mostly for alerting
Evil packets

Traffic Flow

IPS—Intrusion Prevention System
inline network traffic monitoring alerting + ability to drop packets
Alert

Match ?

Traffic Flow

BRKSEC-2006 17796_04_2008_c1

© 2008 Cisco Systems, Inc. All rights reserved.

Cisco Public

47

IDS—Basic Deployment Steps
Analyze Design Deploy Tune
Tuning and management are ongoing

Manage
BRKSEC-2006 17796_04_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public

48

© 2006, Cisco Systems, Inc. All rights reserved. Presentation_ID.scr

24

Enterprise Datacenter IPS/IDS
IDS or IPS?
Can we go inline? If so, where? Datacenter networks typically require high availability Your networking team has likely built in redundancy What failover mechanisms are available in the product?

Availability = MTBF/(MTBF + MTTR) SerialPartsAvailability = ∏(PartAvailability) ParallelPartsAvailability = 1 - [∏(1-PartAvailability)]
CPU 1
PWR 1
Chassis

I/O Card 1

CPU 2

PWR 2

I/O Card 2

Block Diagram of a Simple Redundant System
BRKSEC-2006 17796_04_2008_c1

© 2008 Cisco Systems, Inc. All rights reserved.

Cisco Public

49

Enterprise Datacenter IPS/IDS
IDS or IPS? Availability = MTBF/(MTBF + MTTR)
CPU 1
PWR 1
Chassis

I/O Card 1

CPU 2

PWR 2
Block Diagram of a Simple Redundant System

I/O Card 2

Chassis Availability = 400000/(400000 + 4 hrs) = 0.99999 I/O Card Availability = 200000/(200000 + 2 hrs) = 0.99999 PWR Supply Availability = 500000/(500000 + 2 hrs) = 0.999996 CPU Availability = 100000/(100000 + 2 hrs) = 0.99998 Parallel CPU Availability = 1 - [(1-0.99998) + (1-0.9998)] = 0.9999999996 Parallel PWR Supply Availability =1 - [(1-0.999996) + (1-0.999996) = 0.999999999984 System Availability = Chassis * pCPU * pPWR * Card1 * Card2 = 0.999969999884 [1 - 0.999969999884] x 525,960 Minutes/Yr = 116.76 Minutes
Source: Chris Oggerino, High Availability Network Fundamentals
BRKSEC-2006 17796_04_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public

50

© 2006, Cisco Systems, Inc. All rights reserved. Presentation_ID.scr

25

Enterprise Datacenter IPS/IDS
IDS or IPS? Failover/bypass mechanisms to consider
Fail open vs. fail closed Power failure MTBF for IPS Device
80000 / (80000 + 2) = 0.999975

Redundant DC Network

SNORT IPS Server

0.999994 0.999994 * 0.999975 = 0.999969
BRKSEC-2006 17796_04_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public

0.999975 We Just Lost Our 5th 9!!!
51

Enterprise Datacenter IPS/IDS
Where should we go inline?
Distribution layer aggregation point could be too high of impact Access layer VLAN is typically too much bandwidth

Target a smaller piece of the network containing critical infra
Subnet/VLAN/Segment

IDS or IPS?
IDS HERE

IPS HERE

For more details, see BRKSEC- 3030
BRKSEC-2006 17796_04_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public

52

© 2006, Cisco Systems, Inc. All rights reserved. Presentation_ID.scr

26

Setup IDS
Avoid asymmetry in your traffic view! Minimize the number of platforms and designs
Small DC design, large DC design Distribution layer router uplink traffic ideal

Ingress/ egress traffic only BB uplinks mirrored to sensor

Ingress/ egress traffic only

BB uplinks mirrored to load balancer
© 2008 Cisco Systems, Inc. All rights reserved. Cisco Public

BRKSEC-2006 17796_04_2008_c1

53

Tune IDS
Use Cisco SAFE blueprint Critical to successful deployment Without tuning…
sensors generate alerts on all traffic matching criteria alerts overwhelm monitoring staff NIDS will produce events irrelevant to your environment Will eventually cause NIDS to be ignored or disabled

BRKSEC-2006 17796_04_2008_c1

© 2008 Cisco Systems, Inc. All rights reserved.

Cisco Public

54

© 2006, Cisco Systems, Inc. All rights reserved. Presentation_ID.scr

27

Tune IDS
Run in promiscuous mode
Default configuration Latest signatures

Tune out benign traffic using sensor Tune out benign traffic w/SIM (MARS, etc.)
Start with most frequent alerts Trace to benign source/destination addresses

Create location variables
Demarcations of your network (e.g. DNS, email, DMZ) Elucidates traffic flows Enables more targeted investigations Allows tailoring of filters
BRKSEC-2006 17796_04_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public

55

Tune IDS Using Sensor
Tune out benign traffic using sensor
xxx-dc-ids-1# show stat virt show statistics of the Virtual Sensor Statistics virtual-sensor instance Statistics for Virtual Sensor vs0 <snip> Per-Signature SigEvent count since reset Sig 1101.0 = 22 Sig 3041.0 = 43 Sig 3052.0 = 1 Sig 3135.3 = 13 Sig 3159.0 = 12 Sig 5829.0 = 17 SigID 3030 is TCP Sig 5837.0 = 22 SYN host sweep, Sig 5847.1 = 2 need to look this one Sig 6005.0 = 281 up in MySDN Sig 6054.0 = 49 Sig 6055.0 = 7 Sig 3030.0 = 2045681
BRKSEC-2006 17796_04_2008_c1

© 2008 Cisco Systems, Inc. All rights reserved.

Cisco Public

56

© 2006, Cisco Systems, Inc. All rights reserved. Presentation_ID.scr

28

IDS Tuning
Utilize Cisco’s MySDN service for signature info—mysdn.cisco.com

Benign triggers info tells me that this may be normal traffic

BRKSEC-2006 17796_04_2008_c1

© 2008 Cisco Systems, Inc. All rights reserved.

Cisco Public

57

Tune IDS Using Sensor
Tune out benign traffic using sensor

show me all alarms with sigID 3030 for the past 1 minute

xxx-dc-nms-1# show event alert past 00:01:00 | include 3030 evIdsAlert: eventId=1173255092619515843 severity=high vendor=Cisco originator: hostId: xxx-dc-nms-1 appName: sensorApp time: 2007/04/23 18:50:47 2007/04/23 18:50:47 UTC signature: description=TCP SYN Host Sweep id=3030 version=S2 subsigId: 0 I know my network marsCategory: Info/Misc/Scanner and this is one of our marsCategory: Probe/FromScanner application interfaceGroup: vs0 monitoring devices, I vlan: 0 can tune accordingly participants: attacker: addr: locality=IN 10.6.30.5 target: addr: locality=IN 10.9.4.4 port: 80

BRKSEC-2006 17796_04_2008_c1

© 2008 Cisco Systems, Inc. All rights reserved.

Cisco Public

58

© 2006, Cisco Systems, Inc. All rights reserved. Presentation_ID.scr

29

Tune IDS Using Sensor
Take advantage of network locale variables! We know there are six of these mgmt systems
create a variable
xxx-dc-nms-1# conf t called xxx-dc-nms-1(config)# MGT_SYSTEMS service event-action-rules rules0 variables MGT_SYSTEMS address 10.6.30.5,10.6.30.6,10.30.6.7,10.50.1.5,10.50.1.6,10.50.1.7 create

a filter to drop alerts for multiple signatures filters insert drop_mgt_system_alerts signature-id-range 4003,3030,2100,2152 when the source is attacker-address-range $MGT_SYSTEMS the MGT_SYSTEMS victim-address-range $IN variable actions-to-remove produce-alert|produce-verbose-alert

If a new management server is added, I just modify the network variable and the tuning is done! This can be done with both hosts and networks
BRKSEC-2006 17796_04_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public

59

Tune IDS Using Your SIM
Run reports of most frequent events over 24 hours
Start with the really noisy stuff Get to where you’re not affecting data retention with high volume false positives On-going process: continually verify impact of new signatures

Run reports to find high severity events over 24 hours
Verify alerts have appropriate severity level for your env Modify alert levels based on your policies and network topo i.e. P2P coming from your datacenter should probably be higher than the default “Informational”

BRKSEC-2006 17796_04_2008_c1

© 2008 Cisco Systems, Inc. All rights reserved.

Cisco Public

60

© 2006, Cisco Systems, Inc. All rights reserved. Presentation_ID.scr

30

Custom Signatures
Over 2,000 default signatures in latest pack
Covers thousands of vulnerabilities and odd traffic

Build custom signatures as needed
sig-fidelity-rating 75 sig-description sig-name IRC BOT DOMAINS sig-string-info IRC BOT DOMAINS sig-comment IRC BOT DOMAINS exit engine atomic-ip specify-l4-protocol yes specify-payload-inspection yes regex-string (\x03[Uu][Tt][Kk]\x0b[Tt][Hh][Ii][Rr][Tt][Yy][Ff][Ii][Vv][Ee][Kk]\x03[Oo][Rr][Gg ])|(\x03[Bb][Ll][Aa]\x0f[Gg][Ii][Rr][Ll][Ss][Oo][Nn][Tt][Hh][Ee][Bb][Ll][Oo][Cc] [Kk]\x03[Cc][Oo][Mm]) exit l4-protocol udp specify-dst-port yes dst-port 53

“utk.thirtyfivek.org|bla. girlsontheblock.com”

BRKSEC-2006 17796_04_2008_c1

© 2008 Cisco Systems, Inc. All rights reserved.

Cisco Public

61

Feed NetFlow to SIMs and Other Tools

Feed NetFlow to every tool that will use it
MARS, PeakFlow, etc.

Regionalize deployment
Minimize sending over network
BRKSEC-2006 17796_04_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public

62

© 2006, Cisco Systems, Inc. All rights reserved. Presentation_ID.scr

31

Host Syslog
Capture, store, and relay with syslog-ng For monitoring, be sure your SIM can parse events Key events to log
authentication logs authorization logs (sudo, su, etc.) daemon status logs (know when they stop/start) security application logs (tcpwrappers, EventID portsentry, etc.)
Title User Logon Logon Failure User Logoff Audit Policy Change Audit Log Cleared
63

Windows logging
Agents can relay events via syslog

528 529 - 537

538 Very noisy, grab only important events 612 517
BRKSEC-2006 17796_04_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public

Other Logs
Web server logs
Can verify and elaborate attacks Use HTTP status codes to determine if IDS alert really worked Can provide URL details during attack Apache Send as syslog via httpd.conf setting IIS Send as syslog via MonitorWare Agent

SIM

App server logs
Find way to relay as syslog Send via SNMP events Pull via SQL queries

Oracle logs
Pull logs from AUD$ table via SQL
BRKSEC-2006 17796_04_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public

64

© 2006, Cisco Systems, Inc. All rights reserved. Presentation_ID.scr

32

Perimeter vs. Internal Monitoring

What’s the Difference?
BRKSEC-2006 17796_04_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public

65

Number of Services/Protocols

SMTP DNS FTP HTTP HTTPS SSH POP IMAP ICMP

DMZ Datacenter

SMTP DNS FTP HTTP HTTPS SSH POP IMAP NFS TFTP TACACS TELNET NETBIOS SUNRPC NNTP NTP SQL SNMP LDAP SMB CIFS RPC DCOM NIS DHCP TNS WINS ONS RADIUS HSRP VRRP SCCP SIP H323 Q931 RTP VNC RTSP MSEXCHANGE iSCSI RSYNC LOTUSNOTES RDP HTTP-ALT X11 XDCMP ICMP SQLNET XNS SFTP IPC MSDP LDP VERITAS PXE

Many more false positives sources Tuning more complex
Good relationship with IT application and service owners is key
BRKSEC-2006 17796_04_2008_c1

© 2008 Cisco Systems, Inc. All rights reserved.

Cisco Public

66

© 2006, Cisco Systems, Inc. All rights reserved. Presentation_ID.scr

33

Enterprise Datacenter Monitoring
Complications/Difficulties Traffic: 100+ Gbps globally vs. 4 Gbps outside Protocols: Higher number of services/protocols increases variety and complexity of tuning Alerts: Untuned sensor in large datacenter generates > 100 million alerts/day

BRKSEC-2006 17796_04_2008_c1

© 2008 Cisco Systems, Inc. All rights reserved.

Cisco Public

67

Enterprise Datacenter Monitoring
Complications/Difficulties (Cont.) Higher availability expectations
Enterprise data centers have very high availability requirements Inline “IPS” a hard sell, most hardware not properly redundant We don’t use inline IPS

False positives
Difficult and time consuming to identify Key: good relationship with IT application and service owners

Relatively new technology
Not well understood by IDS and SIMs yet Limited signature base Most signatures based on Internet attacks
BRKSEC-2006 17796_04_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public

68

© 2006, Cisco Systems, Inc. All rights reserved. Presentation_ID.scr

34

False Positives—Examples (Cisco IPS)
SMB: ADMIN$ Hidden Share Access Attempt
An attempt has been made to connect to the hidden windows administration share ADMIN$. This share point does not appear in normal browsing and may access attempts may be an attempt to break into the system.

Windows RPC Race Condition Exploitation
This signature fires when detecting multiple Microsoft DCOM RPC connection attempts in a short period of time. This may be indicative of someone attempting to exploit the race condition that is possible.

Google Appliance ProxyStyleSheet Command Execution
This signature fires when detecting a HTTP request to a Google appliance referring to a remote XSL stylesheet.

Multiple Rapid SSH Connections
This signature fires when there are rapid SSH connection from the same source to the same destination
BRKSEC-2006 17796_04_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public

69

Blanco Wireless: Getting NetFlow
Use a tool such as flow-dscan or nfdump to write an alert to syslog for file transfers sourced from the Oracle10g subnet
flow.acl file uses familiar ACL syntax. create a list named ‘oracle10g’ [mynfchost]$ head flow.acl ip access-list standard oracle10g permit 10.10.0.128 0.0.0.127 concatenate all files from May 5, 2007, filter for src of ‘oracle10g’ network, write the results to syslog

[mynfchost]$ flow-cat /var/local/flows/data/2007-05-05/ft* | flow-filter -Soracle10g | flow-dscan -O 50000000 | logger -f outputfile -p local4 [mynfchost]# crontab -e * * * * * su - netflow -c "env LANG=C; DATE=`date +%Y"/"%m"/"%d`; flow-cat /var/local/flows/data/$DATE/ft* | flow-filter -Soracle10g | flow-dscan -O 500000000 | logger -f outputfile -p local4”

BRKSEC-2006 17796_04_2008_c1

add the command to crontab for automation
© 2008 Cisco Systems, Inc. All rights reserved. Cisco Public

70 70

© 2006, Cisco Systems, Inc. All rights reserved. Presentation_ID.scr

35

Blanco Wireless: Using IDS
Use a custom signature to find cleartext instances of SSN’s between any hosts except your Oracle10g web application servers

custom signature signatures 60001 0 60001 ! “SSN_POLICY” sig-description sig-name SSN_POLICY sig-string-info SSN HANDLING POLICY VIOLATION sig-comment CLEARTEXT SSN exit regex to match SSN engine string-tcp format ###-##-#### event-action produce-verbose-alert specify-min-match-length no regex-string ([0-9][0-9][0-9]-[0-9][0-9]-[0-9][0-9][0-9][0-9]) service-ports 1-65535 exit
BRKSEC-2006 17796_04_2008_c1

© 2008 Cisco Systems, Inc. All rights reserved.

Cisco Public

71 71

Blanco Wireless: Using IDS (Cont.)
Use a custom signature to find cleartext instances of SSN’s between any hosts except your Oracle10g web application servers
xxx-dc-nms-1# conf t xxx-dc-nms-1(config)# service event-action-rules rules0 variables ORACLE_10G address 10.10.0.128-10.10.0.255 variables ORACLE_WEBAPP address 10.10.1.64-10.10.1.95 variables DESKTOP_NETS address 10.10.32.0-10.10.63.255 filters insert drop_desktop_to_webapp_alerts signature-id-range 60001 attacker-address-range $DESKTOP_NETS victim-address-range $ORACLE_WEBAPP actions-to-remove produce-alert|produce-verbose-alert filters insert drop_webapp_to_desktop_alerts signature-id-range 60001 attacker-address-range $ORACLE_WEBAPP victim-address-range $DESKTOP_NETS actions-to-remove produce-alert|produce-verbose-alert
BRKSEC-2006 17796_04_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public

create variables based on IPAM data

apply filters to remove alerting for the oracle web application servers

72 72

© 2006, Cisco Systems, Inc. All rights reserved. Presentation_ID.scr

36

Blanco Wireless: Using IDS (Cont.)
Use a custom signature to find instances of the SQL “describe” command against production database

signatures 60002 0 Custom signature 60002 ! “SQL_describe” sig-description sig-name SQL_DESCRIBE sig-string-info SQL dB enumeration sig-comment Should not see in prod exit regex to match engine string-tcp SQL describe event-action produce-verbose-alert command specify-min-match-length no regex-string [Dd][Ee][Ss][Cc][Rr][Ii][Bb][Ee] service-ports 1433-`433 exit

BRKSEC-2006 17796_04_2008_c1

© 2008 Cisco Systems, Inc. All rights reserved.

Cisco Public

73 73

Blanco Wireless: Using IDS (Cont.)
Use a custom signature to find instances of the SQL “describe” command against production database
apply filters to remove alerting for management systems to dev dBs

filters insert drop_desktop_to_devdb_alerts signature-id-range 60002 attacker-address-range $MGT_SERVERS victim-address-range 10.10.0.120 actions-to-remove produce-alert|produce-verbose-alert

BRKSEC-2006 17796_04_2008_c1

© 2008 Cisco Systems, Inc. All rights reserved.

Cisco Public

74 74

© 2006, Cisco Systems, Inc. All rights reserved. Presentation_ID.scr

37

BRKSEC-2006 17796_04_2008_c1

© 2008 Cisco Systems, Inc. All rights reserved.

Step 6. Maintain and Troubleshoot
Cisco Public

75

Maintain Documented Commitments
Document agreements with support teams
Fixed timelines Expectations (SLAs, OS patching, etc) Refresh commitments every year

Review assets regularly
Look for new assets, new feeds, replaced hosts, etc. Check for feeds/hosts that may have changed/disappeared Check for ownership changes due to re-orgs
BRKSEC-2006 17796_04_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public

76

© 2006, Cisco Systems, Inc. All rights reserved. Presentation_ID.scr

38

Maintain IDS Feeds
Monitor your IDS
Sensor uplinks Sensor processes
This sensor is no longer receiving network traffic!

Watch for spikes/drops in sensor alert volume Have monitoring staff monitor feeds

BRKSEC-2006 17796_04_2008_c1

© 2008 Cisco Systems, Inc. All rights reserved.

Cisco Public

77

Verify Feeds
Syslog feed verification
Script awk to grab hostnames of systems that syslog day and do a diff Ask IT to use a daily cron to re-set syslog.conf on servers

Netflow feed verification
tcpdump -i eth0 port 2060 -c 1000 | grep gw | awk '{print $2}' | sort | uniq

BRKSEC-2006 17796_04_2008_c1

© 2008 Cisco Systems, Inc. All rights reserved.

Cisco Public

78

© 2006, Cisco Systems, Inc. All rights reserved. Presentation_ID.scr

39

Blanco Wireless: Maintenance
Monitor
NetFlow feed IDS feed Syslog feeds Oracle auditing

Maintain agreements with:
DBA team System administrators Network engineers

BRKSEC-2006 17796_04_2008_c1

© 2008 Cisco Systems, Inc. All rights reserved.

Cisco Public

79

Lessons Learned
Start small
Too many events at once is overwhelming Understand/tune each source before adding more Understand “normal” traffic thoroughly before moving on Avoid alerting on false-positives

Use a SIM
Event correlation, false positive reduction

Choose carefully what you want to monitor
…or you’ll waste your time chasing false positives

Use defined playbooks, escalation procedures Have allies in the IT support teams
BRKSEC-2006 17796_04_2008_c1

Network support, DBA’s, webmasters, etc. They can explain/remediate issues you find
© 2008 Cisco Systems, Inc. All rights reserved. Cisco Public

80

© 2006, Cisco Systems, Inc. All rights reserved. Presentation_ID.scr

40

Six Steps to Improve Your Security Monitoring

6. Troubleshoot 5. Feed and tune
4. Choose event sources

3. Select targets 2. Know the network

1. Define your policy

BRKSEC-2006 17796_04_2008_c1

© 2008 Cisco Systems, Inc. All rights reserved.

Cisco Public

81

Q and A

BRKSEC-2006 17796_04_2008_c1

© 2008 Cisco Systems, Inc. All rights reserved.

Cisco Public

82

© 2006, Cisco Systems, Inc. All rights reserved. Presentation_ID.scr

41

Recommended Reading
Continue your Cisco Live learning experience with further reading from Cisco Press Check the Recommended Reading flyer for suggested books

Available Onsite at the Cisco Company Store
BRKSEC-2006 17796_04_2008_c1 © 2008 Cisco Systems, Inc. All rights reserved. Cisco Public

83

Complete Your Online Session Evaluation
Give us your feedback and you could win fabulous prizes. Winners announced daily. Receive 20 Passport points for each session evaluation you complete. Complete your session evaluation online now (open a browser through our wireless network to access our portal) or visit one of the Internet stations throughout the Convention Center.
Don’t forget to activate your Cisco Live virtual account for access to all session material on-demand and return for our live virtual event in October 2008. Go to the Collaboration Zone in World of Solutions or visit www.cisco-live.com.

BRKSEC-2006 17796_04_2008_c1

© 2008 Cisco Systems, Inc. All rights reserved.

Cisco Public

84

© 2006, Cisco Systems, Inc. All rights reserved. Presentation_ID.scr

42

BRKSEC-2006 17796_04_2008_c1

© 2008 Cisco Systems, Inc. All rights reserved.

Cisco Public

85

© 2006, Cisco Systems, Inc. All rights reserved. Presentation_ID.scr

43