D2 - T6 - Logger Lab With Searches Dasboards Reports - 092015

Searches, Dashboards & Reports with Logger*
HP ArcSight Partners Proof of Concept Boot Camp

Technical Day-1
Philippe JOUVELLIER- HP ESP | Global Partner Enablement
philippe.jouvellier@hpe.com
* Lab during this session

© Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.
Typical Logger PoC Use Cases
• What do they need for a Log Management Solution?
• Custom Logs?
• Fast search?
• Strong report capability?
• Custom Dashboards?
• Static correlation using intelligence data?
2 © Copyright 2015 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. HP Restricted. For HP and Partner Internal Use
Before Building Content
• Logger evaluations do tend to revolve around
• ‘High’ inbound event collection
• Easily proven, just apply their event sources
• Seeing ‘their’ data
• If ‘their’ data is standard event sources then use SmartConnectors
• If ‘their’ data is non standard event sources FlexConnectors will be needed
• Attend the HP ESP FlexConnector Training Course
• See ‘Outlining PoC’s’
• Log retention in the long term
• Covered as a capability pre-PoC and technically post-PoC at the sizing stage
• Do use the PoC to gather ‘real’ event rate data, this will assist in accurate sizing
Logger Labs
• Practical exercises will focus on:
#1 Fast searches and visualization
• “Where is my data and what does it look like?”…………………Statistic's and metrics
#2 Dashboards
• “Can I see graphical data ?”…………………………………………….Monitoring activity
#3 Reports
• Customize and run reports…………………………………………......Reporting capabilities
#3 Augment Logger search results with data available in external file
• How to combine intelligence with Logger searches…………….Static correlation
• This applies to standard or custom data
• If you expect to use custom data then attend the FlexConnector Training!
Search
Get familiar with Search
Free text
(no field defined)
Structured
(field defined)
What can I do in a Search?
 Isolate a value or values from either structured or unstructured data

 Aggregate (or combine, group) values and count them
 Graph / Chart values over time or by another field
Popular with customers

| chart count by <fieldname>
| chart count by <fieldname> | sort _count
Field Set: grouping CEF columns into a set
Search Results CEF Field: 1 specific Field from ArcSight schema
CEF Event: a CEF transformed event

Field Sets/Columns
Field Summary: some selected fields
Use Field Summary to show quick data overview
Drilldown in to a field
Use Field Summary to show quick data overview
Drilldown in to a field
Search
Learn to build
your own queries
Use the Demo

script
–It is a good guide
Search
Use the
Advanced
Search
Especially for
structured search
Use the
Demo script
Search
Use the
Advanced
Search
Especially for
structured search
Use the
Demo script
Pipeline Search Operators
Lots of options
The dropdown
help is good!
Pipeline Search Operators
Use Chart to
‘visualise’
searches
Also used with

Look up file
feature
(discussed later)
Again, use the

Demo script!
–It acts as a
tutorial
Top User Accounts Modification by Source User Name
Logger lab #1
Searches
Lab #1: Search and drill-down results
Use Case: Find all configuration modifications applied in the last hour to different devices in a same network.
Then investigate by drilling down configurations modified successfully across results.
What is expected ? A uniform and consistent set of results using HP ArcSight categorization from different log
sources generating different log formats
What content do we need here? collected and parsed events – search feature – drill-down feature
1. From Analyze Search menu window type categoryBehavior = /Modify/Configuration and click on GO
Results show all configuration modification events in the last 10 minutes – we rerun the search on the last hour
2. Select categories from drop down list System Field Sets in upper screen, then click Update Now
3. Click on categoryOutcome (we see now % in Successes, Failures and Attempts)
4. We are interested in successful modifications, therefore Click on Success in selected Fields list
Notice our search being automatically updated
All successful changes applied to different devices in the last hour will show-up
5. Search for any entry where categoryDeviceGroup field/column is populated with Firewall
6. Click on Firewall field/column and notice our search being automatically updated
7. Now on left side of your screen select Name field from Selected Fields list
A list will containing different vendors syntax when same type of event happens
8. Click on « Policy modified » and see the results
Notice
18 the
© Copyright 2015hit number
Hewlett-Packard decreasing
Development asinformation
Company, L.P. The we narrow ouris subject
contained herein search to change without notice. HP Restricted. For HP and Partner Internal Use
9. Add manually | top destinationHostName to your search and click GO

Logger lab #2
Dashboards
Event Data Visualization
Creating Dashboard Panels
First run searches with PIPE operator
Creating Dashboard
Click to “Save the Current Filter”
1- Window will open
(*) When search uses the pipeline operator then Dashboard panel button shows-up
Creating Dashboard
• 2- Select “Dashboard Panel Button”
• Provide a “Panel Title”
• 3-Select “New Saved Search”
• Provide a ‘Saved search name’
• 4-Select ‘New Dashboard’
• Provide a Dashboard name
• 5-Select ‘Panel type’ (chart or table or even both)
• 6-Select ‘Chart type’ (column, Bar, Pie, Area, …)
• 7-Select Chart limit
• 8-Click “Save”
A new One Panel Dashboard was created
Displaying a previously created Dashboard
• Select Dashboard from main menu

• From dropdown list select dashboard name
• Dashboard will show up
Creating Dashboard
• Dashboard with single panel will display

• We can now add 3 more panels to dashboard
Modifying Dashboard
Modifying Dashboard
Click on “Tools”
Select to “Change Layout”
• Drag the panel to new location
• Click “Save”
Dashboard
Looks okay
But there seems to be some ‘null’
field data skewing the graph
Modifying Dashboard
Click on “Configuration”
–Select “Settings”
Select “Saved Search”
Modifying Searches
Select your saved search
Editing Query
Edit the Query
Add the following to the Query:
• AND NOT (destinationUserName IS NULL)
Run Modified Query
Load and run the Query
• Click on the Folder Icon
• Select Saved Searches
• Click on selected saved search
• Click on ‘Load and Close’
• The search is ready to run
Modified Search
Modified Search Results
Advanced Search UI TIP
I used “Advanced Search” under the
! Analyze Tab to determine the search
syntax.
AND NOT (DestinationUserName IS NULL)
Logger lab 2 #: Create Dashboard
Use Case: Create and display a dashboard reflecting login activity during the last day
What is expected? A Dashboard will show up with 5 panels (4 Searches are being created and saved)
What content do we need here? 4 created and saved searches. 1 Dashboard with 5 panels.
Panel#1 : Pie chart with login activity (logins success/failure/attempt)
Panel#2 : Column chart with login successes by User
Panel#3 : Column chart with login failures by User
Panel#4 : Bar Chart with logins by Device Product……….(same search will display panel#4 and panel#5)
Panel#5 : Table with logins by products…………………….. (same search will display panel#4 and panel#5)
• Create the first search with categoryBehavior = /Authentication/Verify

• Add the Pipeline Operator | chart _count by categoryOutcome
• Save the search to a new Dashboard
• Create the second search with categoryBehavior = "/Authentication/Verify" AND categoryOutcome = "/Success" |
chart count by destinationUserName | sort - _count
• Add to the new Dashboard
• Create and save the third search (Failed Logins) and create and save the fourth search (Logins by deviceProduct)
• Don’t forget the fourth search will display results as bar chart and a table (2 panels)
• Edit the Panels and change Chart type and Panel Name.
Logger lab #3
Reports
Reports
• Is Search a primary goal?

• Did the customer like the
pipeline query results?
• Be sure to show the

Export capability
• Select “Export
Results...”
Export results
• Select the settings required

• Click “Export”
• Click “Download Results”
Simple Reports
Reports
Logger Out-Of-the-Box has :
 dozens of reports
 a slew of filters, some field sets
 a few system dashboards
Spend time to understand the categories and general

content
Custom reports take time. Don’t waste time

! reinventing the wheel, use existing reports
to show Logger capability.
Only focus on this if it is required and time permits
Reports
• Repeatable, Structured Reporting
• Logger Reporting works on Structured data only NAVIGATION
Allow users to browse and selected
reports they are interested in
• A Report is built upon a Report Query
– A Query can contain a Parameter (at run-time, specify a value)
• Report(s), Quer(ies), Parameter(s) are packaged into a

.CAB file DESIGN
By selecting Design functions users can
• .CAB files can be imported, exported, with Logger 5.5, create new reports
6.0, even 5.3
• .CAB files are not interchangeable with ESM Reporting;
these are 2 different reporting engines
ADMINISTRATION
Place where users can manage reports
(define types, upload and build
packages, define report parameters, …
Running Defaults Reports
•Find and select a report that
provides expected information
•Select the chosen report

–Click “Run Report” in Actions
section
Creating Reports during PoC
•Find a report that provides similar
information/context
•It is possible to create Logger reports

from scratch
–It is easier to edit existing reports!
•Select the chosen report

–Click “Customize Report”
Creating Reports during PoC
•Save the report under
a different name
Creating Reports
•Select Expand All
•Edit the “Display Fields”
• Provide “Filter Criteria”
• Apply any appropriate

Grouping, Sort Orders etc
• Remember to “Save”
Creating Reports
•Select “Preview”
• Select “Run Now”

 Use desired settings
• Confirm the report is as

required
In many cases this is sufficient to

prove the concept
Creating/Modifying Queries
• In this example we will
modify an existing report
named ‘User Investigation’
• We first run the report
• We are prompted a user

name field
• See the result

Note that the original report had a
! selection field. To remove or edit
this we need to edit the SQL Query
Modifying Queries
Select report
Click on Customize
Select Data Source
Edit Query
Result will
check the Query
Let’s get back to
Properties
Modifying Queries
•Select “Design”
• A new window opens

– This is the window where we edit the query
– Click Edit
– Here we remove the SELECTION FIELD
 Save out of the Query editor
• Click “OK”
The “Result” Tab checks the SQL

Syntax and must be valid before
saving
Reports
The prompt has been
removed
• The report delivers the correct
information
Reports
Logger reports can be highly customised
–Format - Chart/Table
–Query (with SQL Editor)
–Prompt
–Group/Sort/Filter
–Drilldown (to further report)
This brief summary provides the basics

–If reporting is a major requirement prepare in advance!
Logger lab 3 #: Customize and Run Report
Use Case: Modify existing report “Failed Logins by User” and run with saved changes
What is expected? A previously existing report parameters are modified then report is run
What content do we need here? Failed Logins by user report.
1. Select Report from main menu or type Rep in the Take me to windows then click Reports
2. In left Navigation section click Report Explorer
3. Select Default Reports group under Root folder and click on Failed Logins by User report template
4. In Actions section click Customize Report and click Save as – give a meaningful name - and click Save
Report has been copied
5. Click Open (a new window will open) and select your copied report from list (name just provided)
Copied report will open (notice report name shows up on top of windows)
6. Click Expand All
From here you can see/modify the Query, Fields and Filter Criteria's, … and all that applies to this report
7. In Select Display Fields add categoryDeviceGroup to the Selected Fields group
8. In Select Filter Criteria change Count Above 5 to 2, Add filter destinationUserName Is Not NULL, add filter
destinationHostName Is Not Null, add filter destinationAddress Is Not NULL
9. Click Save (copied report has been modified)
10. Click on RUN
Logger lab #4
Look up file (static correlation)

Lookup File
Feature came up with Logger version 6.0
Goal: Augment Logger Search Results with External tables
Looking up a common field that is in both the Logger search results set and the external file
Pull columns from CSV file, use in Logger searches. “Static Correlation”, “Joins in Searches”
• Can add non-event data to Logger Searches
• # Rows x # Fields ~ 5 million
• Chart results
• Redirect to more LOOKUPS or additional searches
• Export results
• For now, LOOKUP tables can only be updated manually
Typical Use Cases:
 Integrate Threat and Intel data
 Integrate Asset Information
 List of Tor Exit Nodes
Look up file anatomy
Lookup files must be in CSV format with the Lookup field names as the first row
First row is treated as

definition of the Field Name 1 Field Name 2 Field Name 3 Field Name …. Field Name N
columns in the table
example an IP
example a string example a string example integer example a string
address
A Lookup
Row 1 mickey 10.0.0.1 bonjour 1111111 ya field is an
Each row in the
table is loaded individual
Row 2 paul 83.214.25.32 bonsoir 1234567890 non column in
sequentially
a lookup
charles 17.1.23.211 hi 333333333 yes file entry
Row …
Row n
louis 62.25.67.111 007 non
Any subsequent row that does not contain the same number of comma-separated values as
the first row will be skipped during the search by the lookup operator*
*If a search using the lookup operator needs to skip one or more rows, a warning message displays on the search page
Look up file – Deep Dive
• Lookup filenames can contain only alphanumeric characters and underscore, and must not begin with a number. Do not include +, -,
or * in the filename. These characters are reserved for the lookup command.
• The maximum size Lookup file that can be uploaded is 50 MB (uncompressed or compressed)
• The maximum disk space allocated for storing Lookup files is 1 GB (cap on overall disk space allowed for storing all Lookup files)
• Maximum number of Lookup entries is 5,000,000. For example, if a Lookup file has four columns and ten rows, the total number of
lookup entries is 4x10=40.
Searches with Look up file - Deep Dive
• When a Lookup file is used in the search, all of its entries will be loaded into memory.
• It is worth noting that the maximum number of rows loaded for lookup varies depending on the
number of columns in the Lookup file.
• For example, if a Lookup file contains 500 columns:
• the maximum number of rows allowed for lookup will be 5,000,000/500 = 10,000 rows
• any subsequent rows will not be used !
• On the other hand, if the table has only 4 columns:
• Then the maximum number rows allowed for lookup will be 5,000,000/4 = 1,250,000 rows
Searches with Look up file Illustrated
search
1-Search using external

table (look up file) is
executed
2-Logger scans events in

CORR engine
3-Results (if any) are
loaded in memory
4-Logger scans entries in
uploaded look up file
5-If value in the Lookup

field is identical with that
in the uploaded Lookup
file then Logger displays
search results
Simple Search with Look up file
Use Case: a search results will

display access failure events
where source IP addresses
match entries in Look Up file
Searches with Look up operator
... | lookup [+/-/*] lookupTableName externalField1 [as loggerField1] [, externalField2 [as
loggerField2] ...] [output [ * | externalField1, externalField2... ] ]
OPERATOR DESCRIPTION
Selects events where the value in the Lookup field
+ (loggerField1, loggerField2) is identical with that in the
uploaded Lookup file (externalField1, externalField2).
Selects events where the value in the Lookup field is not in

- the uploaded Lookup file.
Includes all events regardless of whether they are in the

* uploaded Lookup file.
63 © Copyright 2015 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. HP Restricted. For HP
If you doand
notPartner Internal
specify +, -,Useor *, + is used.
Simple Search with Look up file
Tor Exit Nodes
Download and Import CSV files
https://www.dan.me.uk/tornodes (no more than 30 minutes old)
https://torstatus.blutmagie.de/
Tor Exit Nodes Look Up file
Logger lab 4-a #: Run Search with Lookup File (static correlation)
Use Case: Detect BAD IP addresses inbound connections on port 443 on internal hosts during last 2 hours
What is expected? A search results matching look up file entry(ies) containing BAD IP addresses with details
What content do we need here? A populated Look up file – received events – a search with | lookup operator
1. Select Search from Analyze main menu or type S in the Take me to windows then click Search
2. In Search windows type destinationPort=443 then click GO
Likely tons of results. We need narrowing our search using a look up file named “Malicious_Addresses”
But first see what is inside a lookup file
3. Type Lo in the Take me to windows then click Lookup Files
4. Click Malicious_Addresses and see how lookup file schema was created (columns, data type in each column, …)
Now get back to your search (Search from Main Menu Analyze or type S in the Take me to windows) then click Search
5. Type destinationPort=443 | lookup Malicious_Addresses ip as sourceAddress output * then Click GO
……We ran that same search but we added the lookup feature ( | lookup ) with Malicious_Addresses file.
Search results display now only events where sourceAddresse of events match ip addresses from the lookup file
Logger lab 4-b #: Creating Lookup File
Use Case: Create a new lookup file from suspicious inbound connections on internal hosts
What is expected? A search results exported (CSV format)
What content do we need here? A populated Look up file – received events – a search with | lookup operator
We now want to create a new lookup CSV file from results

1. Type destinationPort=443 | lookup Malicious_Addresses ip as sourceAddress output * | Top ip_Malici
destinationAddress categoryOutcome deviceVendor cat_type_Malici score_Malici
2. When results display click on Export Search Results Icon
3. In Export options select Save to Local Disk – format is CSV and select All Fields if not selected then click Export
4. When done a windows opens, Click on Download Results and save CSV file onto your laptop
5. Now type Lo in the Take me to windows then click Lookup Files then select ADD
6. Give your Lookup file a Name then Browse your laptop then upload the previously saved CSV file
7. When uploaded Click on the lookup file you just created and see the schema
Thank You
Questions ?
© Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. HP Restricted. For HP and Partner Internal Use

D2 - T6 - Logger Lab With Searches Dasboards Reports - 092015

Uploaded by

Document Information

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

D2 - T6 - Logger Lab With Searches Dasboards Reports - 092015

Uploaded by

Copyright:

Available Formats

Searches, Dashboards & Reports with Logger*

HP ArcSight Partners Proof of Concept Boot Camp

* Lab during this session

 Isolate a value or values from either structured or unstructured data

Popular with customers

Search Results CEF Field: 1 specific Field from ArcSight schema

CEF Event: a CEF transformed event

Use the Demo

Also used with

Again, use the

9. Add manually | top destinationHostName to your search and click GO

1- Window will open

• Select Dashboard from main menu

• Dashboard with single panel will display

• Drag the panel to new location

Select “Saved Search”

Add the following to the Query:

• AND NOT (destinationUserName IS NULL)

• Click on the Folder Icon

• Select Saved Searches

• Click on selected saved search

• Click on ‘Load and Close’

• The search is ready to run

AND NOT (DestinationUserName IS NULL)

• Create the first search with categoryBehavior = /Authentication/Verify

• Is Search a primary goal?

• Be sure to show the

• Select the settings required

Spend time to understand the categories and general

Custom reports take time. Don’t waste time

• Report(s), Quer(ies), Parameter(s) are packaged into a

•Select the chosen report

•It is possible to create Logger reports

•Select the chosen report

• Provide “Filter Criteria”

• Apply any appropriate

• Select “Run Now”

• Confirm the report is as

In many cases this is sufficient to

• We first run the report

• We are prompted a user

• See the result

• A new window opens

The “Result” Tab checks the SQL

This brief summary provides the basics

Look up file (static correlation)

First row is treated as

• For example, if a Lookup file contains 500 columns:

• On the other hand, if the table has only 4 columns:

1-Search using external

2-Logger scans events in

5-If value in the Lookup

Use Case: a search results will

Selects events where the value in the Lookup field is not in

Includes all events regardless of whether they are in the

We now want to create a new lookup CSV file from results

You might also like