Course : COMP8025 – Big Data Analytics
Big Data Visualization
Session 12
Dr. Sani M. Isa
This presentation adopted from Raffael Marty, pixlcloud CEO
Overview
• Visualization
• Design Principles
• Dashboards
• SOC Dashboard
• Data Discovery and Exploration
• Data Requirements for Visualization
• Big Data Lake
2 Security. Analytics. Insight.
Why Visualization?
the data... the stats ...
[Link]
5 Security. Analytics. Insight.
Why Visualization?
Human analyst:
• pattern detection
• remembers context
• fantastic intuition
• can predict
[Link]
6 Security. Analytics. Insight.
Visualization To …
Present / Communicate Discover / Explore
7 Security. Analytics. Insight.
Design Principles
Choosing Visualizations
Objective Data Audience
9 Security. Analytics. Insight.
For Example - Lateral Movement
Recon Weaponize Deliver Exploit Install C2 Act
• Objective: Find attackers in the network moving laterally
• Defines data needed (netflow, sflow, …)
• maybe restrict to a network segment
• Audience: security analyst, risk team, …
• Informs how to visualize / present data
10 Security. Analytics. Insight.
Principals of Analytic Design
• Show comparisons, contrasts,
differences
• Show causality, mechanism,
explanation, systematic structure.
• Show multivariate data; that is,
show more than 1 or 2 variables.
by Edward Tufte
11 Security. Analytics. Insight.
Show Context
42
12 Security. Analytics. Insight.
Show Context
42
is just a number
and means nothing without
context
13 Security. Analytics. Insight.
Use Numbers To Highlight Most Important Parts of Data
Summaries
Numbers
15 Security. Analytics. Insight.
Add Context
Additional information about
objects, such as: machine and
• machine user context
• roles
• criticality
• location
user role
• owner machine role
• …
source destination
• user
• roles
• office location
• …
16 Security. Analytics. Insight.
Traffic Flow Analysis With Context
17 Security. Analytics. Insight.
Aesthetics Matter
• Black background
• Blue or green colors
• Glow
[Link]
18 Security. Analytics. Insight.
BORING
19 Security. Analytics. Insight.
Sexier
20 Security. Analytics. Insight.
Dashboard Design Principles
• Audience, audience, audience!
• Comprehensive Information (enough context)
• Highlight important data
• Use graphics when appropriate
• Good choice of graphics and design
• Aesthetically pleasing
• Enough information to decide if action is necessary
• No scrolling
• Real-time vs. batch? (Refresh-rates)
• Clear organization
21 Security. Analytics. Insight.
SOC Dashboards
22
Mostly Blank
23 Security. Analytics. Insight.
Dashboards For Discovery
• Disappears too quickly
• Analysts focus is on their own screens
• SOC dashboard just distracts
• Detailed information not legible
• Put the detailed dashboards on the analysts screens!
24 Security. Analytics. Insight.
Use SOC Dashboard For Context
• Provide analyst with context
• “What else is going on in the environment right now?”
• Bring Into Focus
• Turn something benign into something interesting
• Disprove
• Turn something interesting into something benign
Environment informs detection policies
25 Security. Analytics. Insight.
Show Comparisons
Current Measure
week prior
26 Security. Analytics. Insight.
What To Put on Screens
• News feed summary (FS ISAC feeds, mailinglists, threat feeds)
• Monitoring twitter or IRC for certain activity / keywords
• Volumes or metrics (e.g., #firewall blocks, #IDS alerts, #failed transactions)
• Top N metrics:
• Top 10 suspicious users
• Top 10 servers connecting outbound
Provide context to individual security alerts
[Link]
27 Security. Analytics. Insight.
Data Discovery &
Exploration
28
Visualize Me Lots (>1TB) of Data
29 Security. Analytics. Insight.
Information Visualization Mantra
Overview Zoom / Filter Details on Demand
• summary / aggregation
• data mining
• signal detection (IDS, behavioral, etc.) Principle by Ben Shneiderman
30 Security. Analytics. Insight.
Visualization Challenges
• Access to data
• Parsed data and data context
• Data architecture for central data access and fast queries
• Application of data mining (how?, what?, scalable, …)
• Visualization tools that support
• Complex visual types (||-coordinates, treemaps,
heat maps, link graphs)
• Linked views
• Data mining (clustering, …)
• Collaboration, information sharing
• Visual analytics workflow
31 Security. Analytics. Insight.
