D2 - T3 - Logger Technical Overview - 092015

ArcSight Logger Technical Overview
HP ArcSight Partners Proof of Concept Boot Camp

Technical Day-1
Philippe JOUVELLIER- HP ESP | Global Partner Enablement
philippe.jouvellier@hpe.com
© Copyright 2014 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
Main Objectives
Upon successful completion of this Lab, you will be able to:
• Explain what is HP ArcSight Logger
• Describe Logger architecture
• Under stand the main features and functions of Logger
 Graphical User Interface
 Storage Group
 Searches and Parameters
2 © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
HP ArcSight Logger
(Universal Log Management)
ArcSight Logger is ‘borderless” Universal Log Management solution that can Collect
Everything, Analyze Anything and can be Used Anywhere.
It unifies searching, reporting, alerting and analysis across ANY type of enterprise log
data.
It supports multiple deployment options and can be installed as an appliance and as a

software.
A little Logger history lesson…
• Logger 1.x
• Develop a low cost appliance to receive events at a high event rate
• Why? To compete with product competitors kicking ArcSight ESM’s butt in cost and events per second (EPS).
• We own the logs, we own the customer!, BTW, Logger wasn’t originally intended for a large enterprise customer.
• Logger 2.x
• Reports are added
• No indexing yet, so reports are slow.
• Logger 3.x
• Field-based Indexing is added
• Reports and Field-based Searches are fast now!
• Logger 4.x
• Full-text Indexing is added
• Logger 5.x
• Reporting on unstructured (unparsed/non-normalized) data added through pipeline searches, charts, and dashboards.
• Bloom filters and SuperIndexes
• Logger 6.x
• Speed, Storage size, Map File, New Dashboards, Peering, …
10.4
Universal Log Management
HP ArcSight Logger
NETWORK AUDIT SECURITY HELP DESK APPS INFRASTRUC.
• Collect logs and events data from any log generating source and from anywhere
• Support cyber security, compliance, IT operations, GRC and Log analytics
• Quick Searches + report on years’ of data to investigate outages and incidents quickly and easily
• Cut SAN/storage costs with cheap simple management of huge amounts of log data
5 © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. HP Restricted. For HP and Partner Internal Use
Form Factors
HP ArcSight Logger
• Available as a Software or Hardware Appliance
• Automated enforcement of multiple retention policies
• Static Correlation (LOOKUP file)
• Collection rate was 100k EPS in with small events & no indexing
• Now at least 10k eps in + indexing + searching + reporting
Data Center Appliance Multiple options for

SAN compatible Appliance SMB or Remote Site Appliance
software deployment
Efficient and Intelligent Storage
HP ArcSight Logger
• Flexible storage
 Up to 80 TB/local instance - RAID enabled onboard storage per appliance
 Up to 40 Peer Logger instances configured
• Store years’ worth IT data through a high compression ratios
• Local and external event data archiving
DAS NAS SAN
ARCHIVING
LOCAL BUS
NETWORK
Communications
HP ArcSight Logger
Up to 250 simultaneous HTTPS connections (Web browsers to the Logger Web UI,
Inbound Connections Connectors to SmartMessage receivers and from peer Loggers.
Receivers Receivers listen to events.
A named event source (IP address or hostname of the event sender + name of the
Devices receiver that receives the event)
Used to classify events received from devices. (e.g Device A and Device B events are
Device Groups stored in a group AB and Device C events are stored in group C. Including groups in
searches can limit the data set scanned, thus resulting in faster searches.
Built-in connector forwarding events received to specific destinations such as ESM,

Express, other connectors or other Loggers. Can forward all events or specify filters .
Forwarding rate depends on the complexity of the query (regular expression filter)
Forwarding Connectors used to identify those events transmitted. When no filter is used (all events forwarded)
a Forwarder can do up to 2.5-3K EPS to an ESM destination and 1-3K EPS to connector
destinations.
Event Data Receivers/Forwarders
HP ArcSight Logger
Receivers Forwarders
Log File Syslog Server
Files
Syslog
Syslog
UDP or TCP
Smartconnector
Receiver
ESM
Forwarding
Connector ESM/Express
Smart
Messages
Security Network
Event Sources
Systems
LOGGER
Any Smart/Flex
Connector
Receiver types: Syslog, CEF, SmartMessages, File-based logs, Folder Follower (active text-based logs)
Different forwarder types target different destinations: UDP, TCP, Connector Forwarder, ESM Forwarder, One-time Forwarder with time range
Storage Enforcement
HP ArcSight Logger
Each storage group has a
Event data CORRe retention period defined
and a customizable size
STORAGE RULES Up to 40 storage rules
Up to 6 storage groups GROUP 1 GROUP 6
PRE INSTALLED AVAILABLE

Internal Default
events group
License Enforcement
Product: software version of Logger
License measurement: actual event size and total storage on disk. ;

Mechanism kicks in when for a rolling 30-day window, the daily limit has been exceed more than 5
times. License violation will disable search, reporting and forwarding.
Work flow for measuring event size:

– Raw events: Logger uses raw event size for daily rate monitoring (TCP, UDP, etc. from Receivers)
– CEF events: Original event size is preserved and translated every 5mns through the agent:050 event
Universal Log Management that Scales
HP ArcSight Logger – Peer Networks
• Multiple Loggers work together to support high sustained input rates
Parallel queries can be distributed across Loggers configured in a peer relationship
PEER NETWORK
Logger 1
Logger 2
Reports
Security
Compliance
Logger 3
IT ops Logger 4
Apps
Logger N Searches
Peer Networks Explained
HP ArcSight Logger
LOGGER PEER NETWORK
Events Connector Logger 1
Reports
Security
Events Connector Logger 2
Compliance
IT Opps
Apps
Searches
Events Connector Logger N
peering path
N Loggers is better than 1 Logger!
Math explained
1 x L160GB 4 x L40GB 12 x L15GB

Ingest License 160 GB/day 160 GB/day 180 GB/day
Index rate 10 K eps 40keps 120 K eps
Online search speed 1-3 m eps 9-12m eps 12-36 m eps
Bloom filter scan speed 500mnEPS 2bnEPS 6bnEPS
Search 90 days @ 5keps 77.76s 19.44s 6.48s
Time for 90 day report 10,8 hours 2.7hrs 0,9 hours
Max online storage 80 TB 168TB 960 TB
Days Online @5keps 55 days 220 days 660 days
HP ArcSight Logger
User Interface
Local or external Authentication
ArcSight Logger GUI
Monitoring Logger Processes
Storage Groups Storage Policy
Devices
Device Groups
Storage Rules (up to 40)
Searching All Types of Data
• Example: Internal espionage has been discovered
• Information needs to be gathered quickly to understand what happened, who was
involved, and what the exposure is
• Logger can quickly search both Structured and Unstructured data
Structured Data Unstructured Data

Buildings visited Files uploaded
Cyber investigations
Applications accessed require ALL data sources IM sessions
to determine what’s been
System logins compromised and how Websites visited
Database edits Emails sent/received
CEF data raw syslog data

(from SmartConnectors) (from network devices)
9.24
Google like searches
Auto text typing
Search Results
Appear in a table with self-ranging histogram
• Default: 25 events displayed per page
• Expand/collapse raw event data
9.27
Logger Search Types
Indexed
• Keyword search: Search for full-text keywords (CEF or raw data)
• Field search: Search for specified fields (CEF data only)
• Enable Keyword indexing and/or field indexing for optimum performance
Regex
• Search for regular expression-defined patterns (CEF or raw data)
Pipeline Operators ( | )
• Match patterns of data within search results
• Contain functions that work with either CEF or raw data or both
• Apply additional constraints to narrow search and/or format event display
Boolean Operators (AND, OR, and NOT) are used to connect one or more keyword, field,
and/or regex conditions
9.28
Live Event Viewer
Simple Controls
Adjustable display buffer 20 - 5000 events
(1000 default)
15 minute session limit counter (click to reset)
Reports
Logger Out-Of-the-Box has :
 dozens of reports
 a slew of filters, some field sets
 a few system dashboards
Spend time to understand the categories and general

content
Custom reports take time. Don’t waste time

! reinventing the wheel, use existing reports
to show capability.
Only focus on this if it is required and time permits
Reports
• Repeatable, Structured Reporting
• Logger Reporting works on Structured data only
NAVIGATION
• A Report is built upon a Report Query Allow users to browse and selected
reports they are interested in
– A Query can contain a Parameter (at run-time, specify
a value)
• Report(s), Quer(ies), Parameter(s) are packaged

into a .CAB file
DESIGN
By selecting Design functions users can
• .CAB files can be imported, exported, with Logger create new reports
5.5, 6.0, even 5.3
• .CAB files are not interchangeable with ESM
Reporting; these are 2 different reporting engines
ADMINISTRATION
Place where users can manage reports
(define types, upload and build
packages, define report parameters, …
Data Validation
• DV enables(1) audit-quality validation on Logger
data files.
• Will check the hash value of all data files within

specified time range to validate the data.
• DV process uses SHA1 hash algorithm.
• Compute the hash value for the data files in the specified time range and compares it to the pre-
computed value to determine the integrity of the data file.
• Each data file contains up to 1 GB of data; the hash value is computed once the data file is full.
• If a data file is not full yet, its validation result cannot be computed.
(1)
32 © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. HP Restricted. For HP andavailable to administrators
Partner Internal Use only
Data Validation
Start date for Data Validation

End date for Data Validation
You cannot cancel a Data Validation in progress. The
!
data validation process can take a long time for large
amounts of data. Therefore you should schedule the process
to run during off-peak hours, and narrow down the time
range to include only the data you are interested in.
(1) available to administrators only

Data Validation Results If the system is upgraded to Logger 6.0, data from the
earlier version will have the N/A status. This is because no
! data validation hash value was stored when the data was
created. However, in the case of future upgrades, hash
validation data will be kept, and you will be able to validate
the data after an upgrade.
Displayed Value in
Description
Value Exported File
The hashes match
Intact True
Data is intact
The hashes do not match
Corrupt False
Data has been changed or become corrupt
The file has no hash
Hash Data could not be validated
N/A
unavailable Most likely because data file is not yet full or the data file was
created by an older version of Logger.

Data Validation Results

HP ArcSight Logger
Alerts
Alerts (Real Time / Scheduled) Notification: via e-mail,
SNMP or Syslog, or send to
HP ArcSight Logger ESM/Express
RECEIVERS
Syslog Syslog
Real time alerts

At data processing time
Smart/Flex
SmartMessages
– 25 real time alerts max
Connectors
Scheduled alerts
Log files Files Database searches
- 50 concurrent max
Real time alert name
Regex query
Events to search through
Real time alerts use only

regular expression queries
Scheduled alert name
Query (not on internal events, success on
Authorization/Authentication delete operations)
Events to search through
Match count – number of occurrence's

Time window
HP ArcSight Logger
Installation
Preparing for the Logger Implementation
• Pre-engagement call with the customer.
• Discuss network settings required for Logger
• IP Address, Hostname, DNS
• Duplex settings
• Internal SMTP Mail Relay
• Required ports for access to Logger and Integration with ESM (if needed)
• Use cases for Logger (why did the customer buy Logger?)
• CIPs
• Customization
• Storage Groups
• Retention period of logs
• Integration with Connectors and ESM (if needed)
• Customer needs to download license key for Logger(s)
• Customer is responsible for racking, cabling, and powering the Logger(s)
• IP-based KVM available?

• Not required, but nice to have.
10.41
Initializing Logger Appliance on Network
• Logger requires at least 1 valid IP address - change default using either
• CLI
• Web browser
• Loggers have at least two Network Interface Cards (NICs), identified as eth0 and eth1, that can be
linked to different subnets
• Most Logger appliances have 4 NICs: eth0, eth1, eth2, and eth3
• Configure at least 1 NIC to begin using Logger
2.42
Using the CLI for initial access to Logger Appliance
Using a Web Browser for initial access to Logger
Appliance
• Use a flash-enabled browser (IE 8 or later, or
Firefox 12 or later)
• Login to https://192.168.35.35/
• Default user name: admin
• Default password: password
2.44
Logger Initialization & Configuration Process
• Initialization:
• License
• SAN and SAN Multipathing Configuration (SAN model Loggers only)
• Storage Volume - establish where Logger stores event data
• Storage Groups - apply retention policies to the Storage Volume
• Time Zone, Date and Time Settings
• Index Fields and Full-text Indexing
• System Locale Setting
• Reboot - commit the changes made in previous steps
• Configuration:
• Receivers
• Devices
• Device Groups
• Storage Rules
• Forwarders (if needed)
© Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
45
2.45
SDK API with the HP ArcSigh Logger
• HP ArcSight Logger offers SOAP API for communications.

• Runs with different development language's to communicate with SOAP (Java, Perl,
Python, Ruby, …)
• 3 available Web Services :
• LoginServices
• SearchService
• ReportService
https://loggerip:8443/soap/services/ReportService?wsdl
Enables to communicate with Logger reporting functions as well as authentication.
https://loggerip:8443/soap/services/SearchService?wsdl
Allow events searches.
Thank You
Questions ?
philippe.jouvellier@hpe.com
© Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. HP Restricted. For HP and Partner Internal Use

D2 - T3 - Logger Technical Overview - 092015

Uploaded by

Document Information

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

D2 - T3 - Logger Technical Overview - 092015

Uploaded by

Copyright:

Available Formats

ArcSight Logger Technical Overview

HP ArcSight Partners Proof of Concept Boot Camp

It supports multiple deployment options and can be installed as an appliance and as a

NETWORK AUDIT SECURITY HELP DESK APPS INFRASTRUC.

• Support cyber security, compliance, IT operations, GRC and Log analytics

Data Center Appliance Multiple options for

Receivers Receivers listen to events.

Built-in connector forwarding events received to specific destinations such as ESM,

STORAGE RULES Up to 40 storage rules

Up to 6 storage groups GROUP 1 GROUP 6

PRE INSTALLED AVAILABLE

License measurement: actual event size and total storage on disk. ;

Work flow for measuring event size:

1 x L160GB 4 x L40GB 12 x L15GB

Structured Data Unstructured Data

Database edits Emails sent/received

CEF data raw syslog data

Auto text typing

Spend time to understand the categories and general

Custom reports take time. Don’t waste time

• Report(s), Quer(ies), Parameter(s) are packaged

• Will check the hash value of all data files within

• DV process uses SHA1 hash algorithm.

Start date for Data Validation

You cannot cancel a Data Validation in progress. The

(1) available to administrators only

(1) available to administrators only

(1) available to administrators only

Real time alerts

Events to search through

Real time alerts use only

Events to search through

Match count – number of occurrence's

• Retention period of logs

• Integration with Connectors and ESM (if needed)

• Customer needs to download license key for Logger(s)

• Customer is responsible for racking, cabling, and powering the Logger(s)

• IP-based KVM available?

• HP ArcSight Logger offers SOAP API for communications.

You might also like