Stop The Insanity: Using Event

Correlation Technologies, Tools, and

Techniques to Extract Meaningful
Information from Data Overload
Hugh Njemanze
CTO and Founder

May 2006

© 2005 ArcSight Confidential

Agenda
► What is the Problem?
► What is Correlation?
► How to Think about the Process
► Correlation Technologies, Tools and Techniques
► Benefits of Visual Representations

© 2005 ArcSight Confidential 2

 What is the Problem?

© 2005 ArcSight Confidential 3

What is the Problem?
Complexity of the Security Infrastructure

► Flood of unread data/logs ► Massive false positives

► “Islands” of defense ► Heterogeneous devices

Inefficient and Ineffective

Intrusion
Firewalls Intrusion
Detection
Intrusion Applications
Firewalls Intrusion
Detection Applications
Applications
Firewalls
Firewalls Systems
Detection Applications
Applications Sign-On
Firewalls
Firewalls Detection
Systems
Hosts Applications
Applications Sign-On
Sign-On
Systems
Systems Applications
Applications
Applications
Intrusion Intrusion
Intrusion
Detection
Intrusion Intrusion Directory
Intrusion Detection
Intrusion
Intrusion Anti
Detection
Intrusion
Systems
Detection Detection Anti Services
Detection Systems
Detection
Network
Detection Virus
Systems
Detection
Systems Systems Virus
Databases
Systems Systems
Equipment
Systems
Systems

© 2005 ArcSight Confidential 4

Deal with a Flood of Diverse Data
► Events from many sensors
• NIDS, HIDS, firewalls, anti-virus, more
• Application logs, phone logs, more
► Understanding the protected network
• Vulnerability assessment scanners
• Configuration management databases
► Understanding of vulnerabilities
• CVE
• OASIS

© 2005 ArcSight Confidential 5

The Needle in the Haystack
► Tens of millions
per day
► Millions
► Less than
per day
1 million
per month
► A few thousand
per month

► Raised Alerts
► Case Workflow

9 Attack 9 Verified
9 Pre-attacks formation breaches
9 Normal
9 Audit trail 9 Policy 9 Potential
9 Raw events violations
9 Failed attacks breaches
9 False alarms 9 Identified
vulnerabilities 9 Misuse

© 2005 ArcSight Confidential 6

A Single Integrated Solution
is Required for ESM

ArcSight
Console TM

ArcSight Monitoring, Visualization, and Reporting

ArcSight Real-time Analysis, Correlation, and Workflow

ArcSight Event Collectors

Intrusion
Firewalls Intrusion
Detection
Intrusion Applications
Firewalls Intrusion
Detection Applications
Applications
Firewalls
Firewalls Systems
Detection Applications
Applications Sign-On
Firewalls
Firewalls Detection
Systems
Hosts Applications
Applications Sign-On
Sign-On
Systems
Systems Applications
Applications
Applications
Intrusion Intrusion
Intrusion
Detection
Intrusion Intrusion Directory
Intrusion Detection
Intrusion
Intrusion Anti
Detection
Intrusion
Systems
Detection Detection Anti Services
Detection Systems
Detection
Network
Detection Virus
Systems
Detection
Systems Systems Virus
Databases
Systems Systems
Equipment
Systems
Systems

© 2005 ArcSight Confidential 7

 What is Correlation?

© 2005 ArcSight Confidential 8

What is Correlation?

A relation existing between

phenomena or things which tend
to vary, be associated, or occur
together in a way not expected
on the basis of chance alone.

—Merriam-Webster Dictionary

© 2005 ArcSight Confidential 9

Also, Perhaps, Inference…

The reasoning involved in drawing a

conclusion or making a logical judgment
on the basis of circumstantial evidence
and prior conclusions rather than on the
basis of direct observation.

—Princeton University’s WordNet

© 2005 ArcSight Confidential 10

Highlight Changes in Behavior
► Changes in the typical event flow may indicate
• An ongoing attack
• Denial of service: the source is dead
• Compromise: the source is behaving atypically
► New patterns of behavior may indicate
• The presence of malware
• An insider threat
• Introduction of new software or devices

© 2005 ArcSight Confidential 11

Escalation: Sounding the Alarm
► Generate notifications
• Email, page, pop-up
► Open a case
• Trouble tickets, incident tracking
► Create “alarms”
• Tracking events

© 2005 ArcSight Confidential 12

 How to Think About the
Process

© 2005 ArcSight Confidential 13

Process

► Collection, normalization and aggregation

► Risk-based prioritization with vulnerability and asset information
► Correlation across event sources
• Rule-based correlation
• Statistical Correlation
► Advanced analysis Intelligence

© 2005 ArcSight Confidential 14

 Event Normalization and Categorization
Normalization: Categorization:
Sample Raw Pix Events:
Jun 01 2005 00:00:12: %PIX-3-106011: Deny inbound (No xlate) udp src
outside:10.50.215.97/6346 dst outside:204.110.228.254/6346
Jun 01 2005 00:00:12: %PIX-6-305011: Built dynamic TCP translation from
isp:10.50.107.51/1967 to outside:204.110.228.254/62013
Jun 01 2005 00:00:12: %PIX-6-302013: Built outbound TCP connection
2044303174 for outside:213.189.13.17/80
Jun 02 2005(213.189.13.17/80) to
12:16:03: %PIX-6-106015:
Deny TCP (no connection) from
isp:10.50.107.51/1967 (204.110.228.254/62013)
10.50.215.102/15605
Jun 02 2005 12:16:03: %PIX-6-106015: Deny TCP (no to 204.110.227.16/443
connection) from
flags FIN ACKflags
10.50.215.102/15605 to 204.110.227.16/443 on interface
FIN ACK outside
on interface
outside

© 2005 ArcSight Confidential 15

Diverse Data Sources: Event Normalization
► Comparing apples to apples
• Many vendors
• Many types of sensors
• Lots of overlap
► Normalization
• Common schema: info in the same place
• Categorization: describing the event
• Values: single domain

© 2005 ArcSight Confidential 16

Diverse Data Sources: Event Normalization
► Aggregation: easier to establish equivalence
► Rules can be written once and applied to all sensors
of a given type
► Simplifieslog review when multiple brands of sensor
are present
► Speeds training of new personnel
• Easier to understand events

© 2005 ArcSight Confidential 17

 Risk-based Prioritization

Vulnerability Agents
Scanner
Asset Agent Severity Asset Criticality
Information Mapping of reporting
How important is this
device severity to
asset to the business?
ArcSight severity

Unix/Linux/
AIX/Solaris Severity Model Confidence
Is there a history with Has asset been
Relevance
Security this attacker or target Are ports open on asset?
scanned for open ports
Is it vulnerable?
Device (active lists)? and vulnerabilities?
Agents
Security
Device
Event
Mainframe
& Apps Prioritized
Security Event
Device

Windows
Manager
Systems
© 2005 ArcSight Confidential 18
Event Correlation

► Most overused and least well-defined concept in ESM.

► Combine multiple events through predefined rules
Or analyze statistical properties of event streams
• Across devices
• Heavily utilizing event categorization
► Helps eliminate false positives
► Correlation is not prioritization!
• Can use priorities of individual events

© 2005 ArcSight Confidential 19

Rule-based Correlation

► Combine multiple events through predefined rules

Multiple failed logins

on UNIX systems

Attempted Brute
Force Attack 5 or more failed
Attempted Brute
logins in a minute
Multiple failed logins Force Attack
from same source
on Windows systems

© 2005 ArcSight Confidential 20

Rule-based Correlation

► Combine multiple events through predefined rules

Attempted Brute
Force Attack
Attempted Brute
Force Attack + Successful
Login

Successful
login to Windows system

© 2005 ArcSight Confidential 21

Statistical Correlation

► Analyze statistical properties of event streams

50% increase
in traffic per port
and machine
?

Traffic per port going to 10.0.0.2

► False positives reduction:

• Correlate against other event streams
• Restrict to only monitor specific systems and specific type of traffic

© 2005 ArcSight Confidential 22

j8

Many Correlation/Inference Techniques

► Model-based
• Assets
• Threats
► Heuristic
• Pattern
• Formula
► Mathematical
• Anomaly
• Covariant

© 2005 ArcSight Confidential 23

Slide 23

j8 I think the following slides can be used as some of the voice over from the previous 4-6 slides?
jkyte, 10/11/2005
Model-Based Reasoning
► Checking the protected network
• Does the device exist?
• Applications present
• Operating systems
• Vulnerabilities exposed
• Business significance
► Extensible via active lists
• Attackers: suspicious, recon, hostile
• Devices: scanned, attacked, compromised

© 2005 ArcSight Confidential 24

Heuristic: Formula-Based

Severity
What is the attack potential?

Model Confidence and Relevance

Could it work?

Asset Criticality
How valuable is the target?

Priority
Which incident should be worked first?
© 2005 ArcSight Confidential 25
Mathematical
► Statistical data monitors ► Pattern discovery
• Moving average • Covariant occurrence of
individual events
• Statistics
• Correlation

► Statistics data monitors spot gross changes in the

event flow
• More attacks against certain ports, networks
• Sudden drop in events from a service
► Discovery spots behaviors on the protected network
• New exploits
• Returning exploits: that virus is back!

© 2005 ArcSight Confidential 26

 How: Correlation
Technologies, Tools and
Techniques

© 2005 ArcSight Confidential 27

Traditional Approach – Log Files and Events

© 2005 ArcSight Confidential 28

A Visual Approach
Situational Awareness - Instant Awareness

© 2005 ArcSight Confidential 29

Why a Visual Approach Helps

A picture tells more than a thousand

log lines
► Reduce analysis and response times
• Quickly visualize thousands of events
► Make better decisions
• Situational awareness
• Visualize status of business posture
• Visual display of most important properties
► Be more efficient
• Facilitate communication
• Use graphs to communicate with other teams
• Graphs are easier to understand than textual events
© 2005 ArcSight Confidential 30
Three Aspects of Visual Security Analysis
► Situational Awareness
• What is happening in a specific business area
(e.g., compliance monitoring)
• What is happening on a specific network
• What are certain servers doing
► Real-Time Monitoring and Incident Response
• Capture important activities and take action
• Event Workflow
• Collaboration
► Forensic Investigation
• Selecting arbitrary set of events for investigation
• Understanding big picture
• Analyzing relationships

© 2005 ArcSight Confidential 31

Responding: Monitoring and Reporting
► Live monitoring
• Channels
• Dashboards
► Reporting

© 2005 ArcSight Confidential 32

Situational Awareness – Event Graph Dashboard

© 2005 ArcSight Confidential 33

Real-time Monitoring – Detect Activity

© 2005 ArcSight Confidential 34

Visual Detection
Scanning activity is displayed

Firewall Blocks

Scan Events

© 2005 ArcSight Confidential 35

Visual Investigation

© 2005 ArcSight Confidential 36

Define New Correlation Rules and Filters

1. Rule
Assign for further analysis if
• More than 20 firewall drops
• from an external machine
• to an internal machine
3. Open a ticket for Operations to
quarantine and clean infected machines
2. Filter

• Internal machines on white-list

• connecting to active directory servers

© 2005 ArcSight Confidential 37

Forensic Analysis
► Failed Logins High ratio of failed logins

© 2005 ArcSight Confidential 38

Forensic Analysis
Revenue Generating Systems
Attacks
► Attacks targeting internal systems

© 2005 ArcSight Confidential 39

Summing Up
► Effectivecorrelation enables codifying and leveraging
domain expertise to automate finding the needles in the
haystack of security logs, alerts and events
► Visualization
techniques provide a very intuitive way for
human analysts to quickly spot patterns and activity that
would otherwise be buried in logged data
► Gathering all the data in one place to start with provides
a vantage point from which to apply the tools and
techniques described above

© 2005 ArcSight Confidential 40

 Q&A
Email to: hsn@arcsight.com
© 2005 ArcSight Confidential