ARCSIGHT FLEXCONNECTOR TRAINING

Created by Itachi hasan

WHAT IS AN ARCSIGHT FLEXCONNECTOR

Custom Defined Smart Connector

Collects and normalizes data from unsupported devices.
Fully functioning agent, including categorization, zoning, aggregation,
batching and priority calculation features.

Installed through the ArcSight smart connector installer

Run smart connector installer
Select desired Flex connector type
 TYPES OF FLEXCONNECTOR

Flex Connector Log-file

FlexConnecor Regex log-file
Flex Connector Regex Folder log file
Flex Connector Syslog
Flex Connector Time-based Database
Flex Connector ID-based Database
FlexConnector Multi-Database
FlexConnector SNMP
FlexConnector XML Folder Log file
 FLEX CONNECTOR CONFIGURATION FILE

Configuration file holds the flex parser which will be used to parse the
raw logs
There are 4 steps to creating a FlexConnector configuration file
 Define a parsing mechanism
 Identify and name tokens
 Map tokens to ArcSight schema
 Map device severity to ArcSight severity
 FLEX CONFIGURATION FILE LOCATION
(IMPORTANT)

Base Directory : <Agent Home>/current/user/agent/flexagent/

Log-file : <product_name>.sdkfilereader.properties
Regex log-file: <product_name>.sdkrfilereader.properties
Regex Folder logfile: <product_name>.sdrfilereader.properties
XML Folder logfile: <product_name>.xqueryparser.properties
Time based DB: vendor>/<product_name>.sdktbdatabase.properties
ID Based DB: <vendor>/<product_name>.sdkibdatabase.properties
Multi-DB: <vendor>/<product_name>.sdktbdatabase.properties
Syslog: syslog/<product_name>.subagent.sdkrfilereader.properties
SNMP: <vendor>/sdksnmp.#.snmptrap.rpoperties
FLEX CONNECTOR INSTALLATION
 SELECTING A FLEX
CONNECTOR TYPE

 Log file Flexconnector -For fixed format ,delimited log file (real time log
collection)
 Regex Log file –For variable format log file(real time log collection)
 Regex folder follower- To read logs in batch mode
 Regex Multiple Folder Follower -Read logs from multiple folder (Real
time and Batch mode)
 Time-based Flexconnector -Read event info from tables based on
timestamp value
 ID based Flexconnector -Read event info from tables based on ID value
 SELECTING A FLEX CONNECTOR TYPE

Multiple database Flexconnector – read logs from multiple

database(Time based as well as ID based)
SNMP Connector –Collect logs from snmp traps.
SYSLOG Connector –Security events from syslog packets.
XML Connector –Read logs from XML connector
 EXAMPLES

Log file Example:

01/01/2005-11:33:00,1.1.1.1,52123,2.2.2.2,80,Invalid URL
01/01/2005-12:43:00,3.3.3.3,49123,2.2.2.2,80,Buffer Overflow

Regex Log File

Aug 21 15:28:49 beach sshd[24939]: Failed password for rajiv from 192.168.10.27 port 33654 ssh2
Aug 21 15:28:51 beach sshd[24939]: Accepted password for rajiv from 192.168.10.27 port 33654 ssh2

Time Based Database

 EXAMPLE

ID Based Database

Syslog
My application: Intruder Detected from 1.1.1.1 to 2.2.2.2 High
 SAMPLE CONFIGURATION FILE

 -Simple text file

 Case sensitive
 comments.start.with=#
 delimiter=,
 token.count=5
 token[0].name=Time_of_the_event
 token[0].type=TimeStamp
 token[0].format=yyyy-MM-dd HH:mm:ss
 token[1].name=ClientIp
 token[1].type=IPAddress
 token[2].name=Method
 token[2].type=String
 token[3].name=URL
 token[3].type=String
 token[4].name=Status
 token[4].type=String
 event.deviceReceiptTime=Time_of_the_event
 event.sourceAddress=ClientIp
 event.deviceSeverity=Status
 event.requestUrl=URL
 event.requestMethod=Method
 event.deviceVendor=__getVendor(“MyVendor”)
 event.deviceProduct=__stringConstant(“MyProduct”)
 severity.map.veryhigh.if.deviceSeverity=404,500
 severity.map.medium.if.deviceSeverity=303,302
 CONFIGURATION FILE STRUCTURE

Parsing Mechanism
Token declaration
Event mapping with ArcSight Schema
Severity Mapping
Extra processor
 TOKEN DECLARATION

token. count Number of tokens present in each line of the file

token[x].name User defined name for the tokens
token[x].type Data type of the token
token[x].format Format of the token
 TOKEN TYPES

Integer
Date
IPAddress
Long
MacAddress
RegexToken
String
Time
TimeStamp
 EVENT MAPPING

Mapping the parsed token to ArcSight fields

Type of token must match the ArcSight field.

For Example

event.name=Token1
event.message=Token2
 SEVERITY MAPPING

Assume Token1 values are 23,56,64,54

Example
event.deviceSeverity=Token1
severity.map.veryhigh.if.deviceSeverity=23,54
severity.map.low.if.deviceSeverity=56
severity.map.medium.if.deviceSeverity=64
 CONSOLIDATED CONFIGURATION FILE

 comments.start.with=#
 delimiter=,
 token.count=1
 token[0].name=Time_of_the_event
 token[0].type=TimeStamp
 token[0].format=yyyy-MM-dd HH:mm:ss
 token[1].name=ClientIp
 token[1].type=IPAddress
 token[2].name=Method
 token[2].type=String
 token[3].name=URL
 token[3].type=String
 token[4].name=Status
 token[4].type=String
 event.deviceReceiptTime=Time_of_the_event
 event.sourceAddress=ClientIp
 event.deviceSeverity=Status
 event.requestUrl=URL
 event.requestMethod=Method
 event.deviceVendor=__getVendor(“MyVendor”)
 event.deviceProduct=__stringConstant(“MyProduct”)
 severity.map.veryhigh.if.deviceSeverity=404,500
 severity.map.medium.if.deviceSeverity=303,302
 DATABASE FLEX CONNECTOR

Example
Time based Connector

ID based Connector

CONFIGURATION FILE –TIME BASED
 CONFIGURATION FILE –TIME BASED

 version.order—Specifies the order in which parser files are

executed
 version.query—This property enables you to perform a test
query against the database to validate the database
version.(optional)
 version.id—If the version.query succeeds, the deviceVersion
token is set to the version.id.
 Query-retrieves the rows that were inserted between the last
time query was run and the current time .
 timestamp.field-Specifies the field to use to determine when to
run the next query
 uniqueid.fields-Specifies the field to use to distinguish rows
with the same timestamp field
CONFIGURATION FILE - ID BASED
 CONFIGURATION FILE –ID BASED

 maxid.query -Specifies the query to use to

retrieve the maximum ID present in the database
when the query is run.
 id.field-Specifies the field to use to determine
when to run the next query
 uniqueid.fields-Specifies the field to use to
distinguish rows with the same ID field
 query.limit -Specifies the maximum number of
rows to return when a query is run
REGULAR EXPRESION
CONTINUED
CONTINUED
 SUB MESSAGES

 Example

• Nov 28 22:02:42 10.0.111.2 %PIX-6-106015: Deny TCP (no

connection)from 3.3.3.3/4532 to 4.4.4.4/80 flags RST on interface
outside
• Nov 28 22:06:10 10.0.111.2 %PIX-3-305005: No translation groupfound
for tcp src inside:10.0.112.9/37 dst outside:4.5.6.7/3562
• Nov 29 01:46:42 10.0.111.2 %PIX-6-305005: Translation built for gaddr
1.2.3.4 to laddr 10.0.111.9
• Nov 29 01:35:15 10.0.111.2 %PIX-4-500004: Invalid transport fieldfor
protocol=6, from 2.2.2.2/0 to 3.3.3.3/0
 SUB MESSAGES

 Message divided in two portions.(common to all

messages and one that varies with each message
format)
• Nov 28 22:02:42 10.0.111.2 %PIX-6-106015: Deny TCP (no
connection)from 199.248.65.116/3564 to 10.0.111.22/80 flags RST on
interface outside
 Into:
• Nov 28 22:02:42 10.0.111.2 %PIX-6-106015:
 And:
• Deny TCP (no connection) from 199.248.65.116/3564 to
10.0.111.22/80 flags RST on interface outside
 EXAMPLE

regex=(\S+ \d+ \d+:\d+:\d+) (\S+) %PIX-(\d)-(\d+): (.*)

token.count=5
token[0].name=Timestamp
token[0].type=TimeStamp
token[0].format=MMM dd HH\:mm\:ss
token[1].name=PixIP
token[1].type=IPAddress
token[2].name=PixSeverity
token[2].type=String
token[3].name=SubmessageIdToken
token[3].type=String
token[4].name=SubmessageToken
token[4].type=String
 EXAMPLE CONTINUED

submessage.messageid.token=SubmessageIdToken
-identifies the token that will hold the message identifier

submessage.token=SubmessageToken
-token that containsthe actual sub-message

submessage.count=1
-sub-message IDs (106015)

Submessage[0].messageid=106015
submessage[0].pattern.count=1
submessage[0].pattern[0].regex=Deny (\\S+) \\(no connection\\)
submessage[0].pattern[0].fields=event.transportProtocol
submessage[0].pattern[0].types=String
 CONDITIONAL MAPPING

Event id is 532 type A with parameter 3.3.3.3

Event id is 533 type A with parameter root
Event id is 534 type A with parameter 3.3.3.3

 Scenario:
Event id is 532 or 534, set the ArcSight event field
event.sourceAddress to 3.3.3.3 and if the event id is 533, set
the event.sourceUserName to root.
 EXAMPLE

 regex=Event id is (\\d+) type (\\S+) with parameter (\\S+)

 token.count=3
 token[0].name=EVENTID
 token[1].name=TYPE
 token[2].name=PARAMETER
#Standard mappings
 event.deviceEventClassId=EVENTID
 event.deviceEventCategory=TYPE
#Conditional mappings
 conditionalmap.count=1
 conditionalmap[0].field=event.deviceEventClassId
 conditionalmap[0].mappings.count=2
 conditionalmap[0].mappings[0].values=532,534
 conditionalmap[0].mappings[0].event.sourceAddress=PARAMETER
 conditionalmap[0].mappings[1].values=533
 conditionalmap[0].mappings[1].event.sourceUserName=PARAMETER
 EXAMPLE 2

 conditionalmap.count=2
 conditionalmap[0].field=event.deviceCustomString3
 conditionalmap[0].mappings.count=2
 conditionalmap[0].mappings[0].values=04
 conditionalmap[0].mappings[0].event.deviceEventClassId=__stringConstant("Posted")
 conditionalmap[0].mappings[1].values=03
 conditionalmap[0].mappings[1].event.deviceEventClassId=__stringConstant("Pending")

 conditionalmap[1].field=event.deviceAction
 conditionalmap[1].mappings.count=2
 conditionalmap[1].mappings[0].values=01
 conditionalmap[1].mappings[0].event.name=__stringConstant("Internal Account
Transfer")
 conditionalmap[1].mappings[1].values=02
 conditionalmap[1].mappings[1].event.name=__stringConstant("External Account
Transfer")
 EXTRA PROCESSOR

To chain two configuration files together

Useful if you need to use two or more different types of FlexConnectors
for the same data
Regular expression to parse data that was obtained from a time-based
SQL database.
Configuration files need to be placed in the \user\agent\flexagent folder
 EXAMPLE

extraprocessor.count=1
extraprocessor[0].type=regex
extraprocessor[0].filename=netiq/netaiq
extraprocessor[0].field=event.message
extraprocessor[0].flexagent=true
extraprocessor[0].clearfieldafterparsing=false
EXTRA PROCESSOR TYPE
 EXAMPLE

extraprocessor.count=2

extraprocessor[0].type=regex
extraprocessor[0].field=event.message
extraprocessor[0].filename=DataSetParser
extraprocessor[0].clearfieldafterparsing=false
extraprocessor[0].flexagent=true

extraprocessor[1].type=regex
extraprocessor[1].field=event.message
extraprocessor[1].filename=DataSetParser1
extraprocessor[1].clearfieldafterparsing=false
extraprocessor[1].flexagent=true
 MULTILINE REGEX

To support multi-line messages, we need to define the message start and
end in the configuration file
 SAMPLE MULTILINE LOG FILE

|01/01/2005 11:00:50|1.1.1.1|7663|2.2.2.2|80|this
is
a
message
that
takes
multiple
lines|
01/01/2005 11:00:51|1.1.1.1|7663|2.2.2.2|80|this
is another large message that takes
multiple lines|
 SAMPLE

multiline.starts.regex=\|\d+/\d+/\d+ \d+:\d+:\d+\|.*
multiline.ends.regex=.*\|$

Output:-

|01/01/200511:00:50|1.1.1.1|7663|2.2.2.2|8|this is a message that takes

multiple lines|
 MULTILINE REGEX CONFIGURATION FILE

multiline.starts.regex=\\|\\d+/\\d+/\\d+
\\d+\:\\d+\:\\d+\\|.*
regex=\\|(.*?)\\|(\\S+)\\|(\\d+)\\|(\\S+)\\|(\\d+)\\|(.*)|
token.count=6
token[0].name=Timestamp
token[0].type=TimeStamp
token[0].format=MM/dd/yyyy HH\:mm\:ss
token[1].name=SourceAddress
token[1].type=IPAddress
token[2].name=SourcePort
 FUTURE CONCEPTS

Advanced Regex Usages.

Parser Overrides.
Extra Mapping Files.
Agent Property file Important Configurations.
Details on Flex Connector Types.
Basic Troubleshooting.
Lab Exercises for Practice using Regex.
Q & A.
Open Suggestions.
END