Splunk 201 - Threat Hunting Workshop

© 2017 SPLUNK INC.
Splunk 201 –
Threat Hunting Workshop
Based on BOTS 1.0 Dataset
Paul Pang
Jun 2018 | Version 1.1

© 2017 SPLUNK INC.
How to access the Lab

Make sure you are accessing to the
WiFi AP
- SSID : Splunk-Guest-HK
You can download this
- WPA password : 609Nathan PPT and csv from
https://splunk.box.com/v/sec201
Use your browser to access the Password : splunkpaul
following site :
– http://192.168.70.138:8000
– http://192.168.70.175:8000
– User : user01 .. user30

– Password : changeme
© 2017 SPLUNK INC.
How to access the Lab

Make sure you are accessing to the
WiFi AP
- SSID : deception-h@cker You can download this
- WPA password : @utobot$rollout$ PPT and csv from
https://splunk.box.com/v/sec201
Password : splunkpaul
Use your browser to access the
following site :
– http://192.168.10.121:8000
– User : user01 .. user30

– Password : changeme
© 2017 SPLUNK INC.
#whoami
10 Years as Splunk Ninja
50+ BOTS @APAC
20+ Years as SOC Advisor

© 2017 SPLUNK INC.
Agenda
▶ Splunk Basic
▶ ex1 : Web Defacement
▶ ex2 : Threat Intelligence
▶ ex3 : Hunting Known Threat : Crypto Miner, Botnet and Malware
▶ ex4 : Hunting Unknown Threat : DNS tunneling & exfiltration
▶ Latest APT operations and updates
© 2017 SPLUNK INC.
Organization of the SOC/CSIRT Team
Source : SANS, Building a world class security operation center roadmap 2015
6
© 2017 SPLUNK INC.
Threat Hunting Techniques
Pyramid of Pain
Diamond Model
Cyber Kill Chain : APT Detection and Response
© 2017 SPLUNK INC.
Gain Access Create additional Conduct

Transaction to system environment Business
Web
Threat intelligence
.pdf
Portal
Network
Activity/Security
Proxy log
MAIL
C2 WEB
communication
to blacklist
Events that
contain link to file
Host What created the How was

program/process process
Activity/Security ? started?
Process
making C2
traffic
.pdf Calc.exe Svchost.exe
Auth - User Roles

8
© 2017 SPLUNK INC.
Data Source Mapping
Web Email Endpoint Proxy/DNS
Command
Recon Weaponize Deliver Exploit Install & Action
Control
CMDB and Threat Intelligence

© 2017 SPLUNK INC.
Splunk Security Portfolio
DATA PLATFORM ANALYTICS OPERATIONS
> Universal indexing ES CONTENT

UPDATE
> Petabyte scale
> Multi-schema
> Search, alert, report, visualize
> Broad support
Machine Learning Toolkit

(MLTK)
© 2018 SPLUNK INC.
Splunk Security Portfolio
ANALYTICS OPERATIONS
DATA
PLATFORM
Platform for Machine Data
© 2017 SPLUNK INC.
Search All Your Machine Data
▶ Real-time and historical data Search all

on-premises, in the cloud or both your data
▶ Over 140 commands including

Results right away
anomaly detection and Schema-on-the-fly
machine learning
Data Real-Time
Search Process
Monitor Parsing Pipeline Real-Time
Parsing Queue
Input
Index Queue
• Source, event typing Buffer
TCP/UDP • Character set
normalization
Input • Line breaking
• Timestamp
Scripted identification
Indexing
• Regex transforms Pipeline Raw Data
Input Index Files Splunk
Index
© 2017 SPLUNK INC.
Schema-on-the-Fly
Raw events
Auto-detected fields and values

© 2017 SPLUNK INC.
Extract Fields Anytime
▶ Simple field extraction

• Highlight-to-extract multiple fields at
once
• Apply keyword
search filters
• Specify required text
in extractions
• View diverse and
rare events
• Validate extracted
values with field stats
© 2017 SPLUNK INC.
Splunk Commands Used
▶ metadata ▶ timechart
▶ AND OR NOT ▶ sort
▶ stats –count, values, average, AS ▶ reverse
▶ lookup ▶ transaction
▶ eval – lower, length, round ▶ rex
▶ fields ▶ search
▶ table ▶ iplocation
▶ head ▶ geostats
▶ tail
© 2017 SPLUNK INC.
Ex 1 : Using Splunk for Security

Investigation the Web Defacement
© 2017 SPLUNK INC.
Luckily Everything Is Captured In Splunk
▶ Microsoft Sysmon
▶ Windows Events
▶ Windows Registry
▶ IIS
▶ Splunk Stream (wire data)
▶ Suricata
▶ Fortigate (NGFW)
© 2017 SPLUNK INC.
Reviewing All The Data Available

Search UI
© 2017 SPLUNK INC.
Hands-on Exercise
Selected and Interesting Fields
Question 1
▶ What is the likely IP address of someone scanning imreallynotbatman.com for

web application vulnerabilities?
Answer : 40.80.148.42
© 2017 SPLUNK INC.
Search Basics
▶ What is the likely IP address of someone scanning imreallynotbatman.com for
web application vulnerabilities?
© 2017 SPLUNK INC.

© 2017 SPLUNK INC.
Default Selected Fields

© 2017 SPLUNK INC.
More Fields
© 2017 SPLUNK INC.
Histogram
© 2017 SPLUNK INC.
Time Picker
© 2017 SPLUNK INC.
SPL Help
© 2017 SPLUNK INC.
Autocomplete
© 2017 SPLUNK INC.
Fast v Smart v Verbose Mode

Search
▶ Different modes for searching
▶ Trade off of speed v fidelity
▶ Can toggle between them
▶ Search speed implications
▶ Good to start with Smart and then go from there
© 2017 SPLUNK INC.
Smart Mode
Default
© 2017 SPLUNK INC.
Smart Mode
With stats
© 2017 SPLUNK INC.
Fast Mode
© 2017 SPLUNK INC.
Fast Mode with Statistics

© 2017 SPLUNK INC.
Verbose Mode
© 2017 SPLUNK INC.
Verbose Mode
With stats
© 2017 SPLUNK INC.
Search Basics
Other Controls
© 2017 SPLUNK INC.
Field Detail
© 2017 SPLUNK INC.
Implicit AND in Search
AND AND
© 2017 SPLUNK INC.
Parenthesis
© 2017 SPLUNK INC.
NOT
© 2017 SPLUNK INC.
Equivalent Searches
Assuming Four sourcetypes
AND AND
AND AND
© 2017 SPLUNK INC.
What is the likely IP address scanning

imreallynotbatman.com for web application
vulnerabilities?
▶ index=botsv1 imreallynotbatman.com sourcetype=stream:http
© 2017 SPLUNK INC.
Hands-on Exercise
Question 2
▶ What are the top URIs being returned during the scan on
imreallynotbatman.com?
Answer : /joomla/index.php/component/search/
© 2017 SPLUNK INC.
Adding Transformational Search Commands
▶ Search commands are separated by |

▶ String multiple search commands together
▶ Transformation search commands take search results and then manipulate those
results
▶ What are the top URIs being returned during the scan on
imreallynotbatman.com?
© 2017 SPLUNK INC.
Building on Our Previous Search

© 2017 SPLUNK INC.
Introducing the stats command

And its friend sort
▶ stats
• Provides a way to calculate values based on result sets
• Sum, average, count, distinct count, standard deviation, values and more...
• Provides a way to group these calculations by other fields
• Provides a way to rename the resulting value that is calculated
▶ sort
• Orders in ascending or descending order based on the specified field(s)
Sort descending by count so the greatest values come to the top and display the top 10
Generate a count by URI to see what values are most frequently seen
© 2017 SPLUNK INC.
Our Results?
© 2017 SPLUNK INC.
Hands-on Exercise
Question 3
▶ What is the IP address of the server hosting imreallynotbatman.com?
Answer : 192.168.250.70
© 2017 SPLUNK INC.
Finding the Server IP

© 2017 SPLUNK INC.
Hands-on Exercise
Question 4
▶ What IP address is likely attempting a brute force password attack against

imnotreallybatman.com?
Answer : 23.22.63.114
© 2017 SPLUNK INC.
HTTP Get v Post Traffic Flow
▶ HTTP GET will generally retrieve data

GET
File x
imreallynotbatman.com
▶ HTTP POST submits data to be processed
POST
File x
© 2017 SPLUNK INC.
Wildcards
▶ Wildcards allow us to only have a portion of a value and still search on it

▶ Structured
• sourcetype=stream:*
▶ Unstructured
• *.exe

© 2017 SPLUNK INC.
Wildcard Search
© 2017 SPLUNK INC.
Our Search...
© 2017 SPLUNK INC.
table Command
© 2017 SPLUNK INC.
stats command for the win...

Answer : 23.22.63.114
© 2017 SPLUNK INC.
Hands-on Exercise
Question 5
▶ What company created the web vulnerability scanner used by Po1s0n1vy?

Type the company name. (For example "Microsoft" or "Oracle")?
Answer : Acunetix
© 2017 SPLUNK INC.
Check the HTTP header payload
▶ What company created the web vulnerability scanner used by Po1s0n1vy? Type
the company name. (For example "Microsoft" or "Oracle")?
© 2017 SPLUNK INC.
You can find this from the stream:http

src_headers
© 2017 SPLUNK INC.
You can also found the similar information from

http_user_agent field
© 2017 SPLUNK INC.
What If You Don’t Know What Acunetix Is?
▶ If you see something in the logs

you don’t understand, Google it!
© 2017 SPLUNK INC.
At the halfway mark...
▶ Splunk Basics
▶ UI Familiarization
▶ Searching with structured fields and unstructured values
▶ Compound Searches (AND OR NOT)
▶ Introduction of stats, sort, table commands
▶ Wildcards
© 2017 SPLUNK INC.
Break
© 2017 SPLUNK INC.
Hands-on Exercise
Question 6
▶ What was the first brute force password used?
Answer : 12345678
© 2017 SPLUNK INC.
What was the first brute force password used?

reverse Command
▶ Building on the basic search
© 2017 SPLUNK INC.
Newest at the Top

© 2017 SPLUNK INC.
Add reverse Command
Password
© 2017 SPLUNK INC.
Extracting the Password from the String

rex Command
© 2017 SPLUNK INC.
search Command
▶ The search command is implied at the start of every search

▶ Search command can be used after doing transformational searches to narrow
results
▶ Search with =* will return all rows that have ANY value
▶ Good way to return NOT NULL
© 2017 SPLUNK INC.
head & tail Commands
▶ Returns the first (or last) N number of specified results in search order
▶ Opposite of head is tail

© 2017 SPLUNK INC.
Putting it all together
▶ What was the first brute force password used?

• 12345678
© 2017 SPLUNK INC.
Hands-on Exercise
Question 7
▶ What was the correct password for admin access to the content management
system running "imreallynotbatman.com” ?
Answer : batman
© 2017 SPLUNK INC.
stats Isn’t Just For Counting
▶ Were any passwords used more than once and if so, where did they login from?
▶ stats – Calculate count, sum, standard deviation, average and more

• Can also return values and calculation at the same time
© 2017 SPLUNK INC.
stats Command Provides Different Views

© 2017 SPLUNK INC.
Hands-on Exercise
Question 8
▶ What was the average password length used in the password brute forcing
attempt, rounded to the nearest integer?
Answer : 6
© 2017 SPLUNK INC.
Calculating an Average and Rounding the Result
© 2017 SPLUNK INC.
eval and stats Command

Beautiful Together
▶ Calculate the length of a string (userpassword) using the eval command
▶ Calculate the average of all the lengths using stats and rename it to avgPword
▶ Use the eval command with the round function
Calculate a length for the Calculate the average of all Round the avgPword field to 0
userpassword string and store lenpword and rename it decimal places and put it into
the value in lenpword avgPword the avgPword field
© 2017 SPLUNK INC.
Calculating an Average and Rounding the Result
© 2017 SPLUNK INC.
Visualizing The Frequency of the

Brute Force Attack
▶ | timechart
• Generates a table of summary statistics
• Can be formatted as a visualization
• Display statistical trends over time
• X-axis represents time and Y-axis can be any field value, count of values, or statistical
calculation
▶ If we wanted to see the frequency of the brute force attacks over time, timechart
can help us visualize this
© 2017 SPLUNK INC.
Brute Force Example
▶ Leverage our earlier extraction
▶ timechart syntax is similar to stats

• Count grouped by the destination address
• Span is set to bucket the data in the time chart by a range, could be 1 sec, 1 min, 1 hour, 1 day
or some other interval
© 2017 SPLUNK INC.
When span is 1 second

© 2017 SPLUNK INC.
And it’s associated chart

© 2017 SPLUNK INC.
When span is 1 minute

© 2017 SPLUNK INC.
And it’s associated chart

© 2017 SPLUNK INC.
Timechart Command
Which Span is Better?
▶ It depends....
© 2017 SPLUNK INC.
Hands-on Exercise
Question 9
▶ How many seconds elapsed between the time we first saw the password batman
and the time we saw it again?
Answer : 92.17
© 2017 SPLUNK INC.
Calculating the Time Difference Between Multiple

Events
▶ Same extraction as before, this time let’s search for the string batman and table
the output with the time and src address
© 2017 SPLUNK INC.
transaction command
▶ Use transaction command to group these events together

▶ Output the duration value which was created with the transaction command
Group events together based on common Duration is a value created with transaction
value(s). In this case group events as a that calculates the difference between the
transaction based on the same userpassword first and last event in seconds
© 2017 SPLUNK INC.
transaction Command
© 2017 SPLUNK INC.
Hands-on Exercise
Question 10
▶ What is the name of the file that defaced the imreallynotbatman.com website?
(For example "notepad.exe" or "favicon.ico") ?
Answer : poisonivy-is-coming-for-you-batman.jpeg
© 2017 SPLUNK INC.
HTTP Get v Post Traffic Flow
▶ HTTP GET will generally retrieve data

GET
File x
imreallynotbatman.com
▶ HTTP POST submits data to be processed
POST
File x
© 2017 SPLUNK INC.
Web Server Traffic
▶ How often should web servers start a conversation?

▶ If they are starting the conversation is that interesting and should be looked at?
▶ Very few results in HTTP when the Web Server is the source
© 2017 SPLUNK INC.
Anything in Common - stream:http and suricata?

© 2017 SPLUNK INC.
What Do the Firewall Logs Say?

© 2017 SPLUNK INC.
Which Search Should I Start With?

© 2017 SPLUNK INC.
Use A Web Site Categorization Technology to Filter

© 2017 SPLUNK INC.
The Firewall Confirms stream:http and suricata

© 2017 SPLUNK INC.
APT Threat Picture

Vulnerability Scan
Acunetix
IP: 40.80.148.42
GET imreallynotbatman.com
poisonivy-is-coming-for-you-batman.jpeg IP: 192.168.250.70
IP: 23.22.63.114
© 2017 SPLUNK INC.
IP Location
▶ Splunk ships with a copy of GeoLite2-City.mmdb from Maxmind

▶ iplocation Command will enhance an event IP address with City, State, Country,
Latitude and Longitude
© 2017 SPLUNK INC.
Extending IP Location to Visualizations
▶ With geostats and iplocation events can be easily laid out on a map
▶ Maps can be drilled down to a point, additional configuration can be done to drill
down to the street level
© 2017 SPLUNK INC.
Final
▶ head & tail

▶ rex – Search time field extractions
▶ eval and more stats
▶ timechart
▶ transaction
▶ iplocation and geostats
FULL TIME
90:00
© 2017 SPLUNK INC.
Other Web Attack Example : Apache Struts 2

Exploit Analysis
Apache Struts 2 0-Day vulnerability (CVE-2017-5638), was recently discovered
on March 6, 2017
Equifax confirms Apache Struts

https://www.solutionary.com/resource-
security flaw it failed to patch is to
center/blog/2017/03/apache-struts-2-exploit-analysis/ blame for hack
10
1
© 2017 SPLUNK INC.
How to hunt ?
sourcetype=weblog *.apache.struts2* wget OR “service iptables” OR chmod
10
2
© 2017 SPLUNK INC.
Detect ShellShock exploit attempts
http://security.stackexchange.com/questions/68327/what-do-shellshock-attacks-look-
like-in-system-logs
10
3
© 2017 SPLUNK INC.
search for unix similar expressions like... chmod

777,echo, ls, cd etc.
sourcetype=weblog chmod 777 OR ls OR cd OR “/bin”
10
4
Free for 1st Year Subscription
© 2017 SPLUNK INC.
▶ Guidance on how to analyze,

investigators, and respond to
threats
▶ Structure to connect the dots
from the data-sources to detect,
investigate, remediate and
monitor for attacks
▶ Focus on malware, vulnerability
and user behavior
• Cyber Kill-Chain (Lockheed
Martin) and ATT&CK (MITRE)
models to aid analysis
• Regular updates
© 2017 SPLUNK INC.
Scenario-based Knowledge for your L3 team
10
6
© 2017 SPLUNK INC.
Threat Intelligence
(IOC) Analysis
© 2017 SPLUNK INC.
Threat Intelligence Report

http://blog.shadowserver.org/2015/08/10/the-italian-connection-an-analysis-of-exploit-
supply-chains-and-digital-quartermasters/
10
8
Threat Data/Raw IOCs
© 2017 SPLUNK INC.
10
9
Threat Data from Mandiant APT1
© 2017 SPLUNK INC.
11
0
© 2017 SPLUNK INC.
© 2017 SPLUNK INC.
Hands-on Exercise
Question 11
▶ Based on the data gathered from this attack and common open source
intelligence sources for domain names, what is the email address that is most
likely associated with Po1s0n1vy APT group ?
Answer : lillian.rose@po1s0n1vy.com
© 2017 SPLUNK INC.
Question 107
Beginner
▶ Based on the data gathered from this attack and common open source
intelligence sources for domain names, what is the email address that is most
likely associated with Po1s0n1vy APT group?
▶ Use a site like www.robtex.com, whois.domaintools.com, or dnswhois.info to

pivot on the observed attacker domain name to other domain names associated
with the same IP address.
▶ Examine whois records for suspect email addresses.

© 2017 SPLUNK INC.
Back to VirusTotal
© 2017 SPLUNK INC.
Continuing with VT
▶ You can lie a lot on your DNS

information but you still need an
email address....
© 2017 SPLUNK INC.
DomainTools
whois.domaintools.com
© 2017 SPLUNK INC.
Reverse Whois
▶ Additional cost services

but perhaps your
organization has them
© 2017 SPLUNK INC.
Validation
© 2017 SPLUNK INC.
Correlate with Lookup

It is useful to correlate the info in firewall log with external threat intelligence, and
estimate if there is potential missed risk not falling inside the radar
Flow
– Create the external threat intelligence lookup table
– Search the dest_ip that match the list but the action is allowed
– Report the result in table format and the information in the threat intelligence lookup table
11
9
© 2017 SPLUNK INC.
Lookups
Threat Data ( sample file : threat_list.csv )
© 2017 SPLUNK INC.
12
1
© 2017 SPLUNK INC.
Step 1 : Import the Lookup File
12
2
© 2017 SPLUNK INC.
Threat Intelligence
In reality, the threat intelligence can be periodically pulled into Splunk lookup
using scheduled script
– e.g. Tor servers, known C&C servers,…
Pre-configured in Enterprise Security App
(http://docs.splunk.com/Documentation/ES/latest/User/ThreatIntelligence)
In this exercise, a static mock-up lookup is used
– Download the threat_list.csv from
ê https://splunk.box.com/s/uankij3knyqads1xanfawd5y0wrlkluf
– Import the Lookup file into Splunk
ê Setting à Lookups à Lookup table files
– Define the lookup definition
ê Setting à Lookups à Lookup definitions
12
3
© 2017 SPLUNK INC.
Threat Intelligence Lookup

Example only threat_ip,threat_type
95.140.2.144, Malware
2 columns, 2.239.94.219, Malware
– threat_ip: lookup key, to match dest_ip 148.183.2.168, C&C
– threat_type: description about the threat type 97.48.2.191, Malware
223.2.75.183, C&C
76.2.20.172, Malware
2.95.46.113, C&C
196.2.238.251, RAT
36.2.121.136, Malware
2.10.92.183, RAT
12
4
© 2017 SPLUNK INC.
Step 2: Define the Lookup Definition
12
5
© 2017 SPLUNK INC.
Step 3 : Verify the Lookup table is OK

| inputlook threatLookup
12
6
© 2017 SPLUNK INC.
Step 4 : Correlate External Threat

Looking for events that dest_ip matches the threat_ip in the lookup file,
as well as action is 'allowed', return the result in table format, with time,
client ip, destination ip and threat type
sourcetype=cisco:asa | lookup threatLookup

threat_ip as dest_ip | search threat_type=* | table _time, host,
src_ip,dest_ip,threat_type lookup definition
'input' field in events
'input' field in lookup definition 'output' field in lookup definition

12
7
© 2017 SPLUNK INC.
Search and Visualize
12
8
ES’ Threat Intelligence – Pre-config Threat Lists
© 2017 SPLUNK INC.
129
Customer Case: Client running P2P ( BT bit torrent )
© 2017 SPLUNK INC.
Client IP : 172.26.228.230
Time : 18:10 5/3/14
Threats :
Accessing following Bad IP
- Tor (anonymous proxy)

- Piratebay (BT host)
- Blocked IP site
- Known spyware site
Verified with PC configuratio

and this PC has installed th
BT client software.
130
Detect Crypto Miner ( Browser OR Applications)
© 2017 SPLUNK INC.
Execute particular command line

Variations in the parameter names by variations of mining software
13
1
© 2017 SPLUNK INC.
Well-known mining pool sites

SANS report CoinBlockerLists
https://www.sans.org/reading-
room/whitepapers/threats/detecting- https://github.com/ZeroDot1/CoinBlockerLi
crypto-currency-mining-corporate-
environments-35722 13
2
© 2017 SPLUNK INC.
Add Crypto Currency Threat Intelligence

prevent cryptomining in the browser or other applications
Crypto Miner Tracker
https://zerodot1.gitlab.io/CoinBlockerListsWeb/inde
x.html
1) Domain:
https://zerodot1.gitlab.io/CoinBlockerLists/list.txt
2) ignoring regex :
(^#|^\s*$)
3) fields :
domain:$1,description:Cryptomining_domain
13
3
© 2017 SPLUNK INC.
Add Crypto Currency Threat Intelligence

prevent cryptomining in the browser or other applications
1) MiningServerIPs:
https://zerodot1.gitlab.io/CoinBlockerLists/
MiningServerIPList.txt
2) ignoring regex :
(^#|^\s*$)
3) fields :
ip:$1,description:Cryptomining_IP
13
4
© 2017 SPLUNK INC.
© 2017 SPLUNK INC.
Add Ransomware Threat Intelligence

detect active ransomware in the browser or other applications
Ransomware Tracker
https://ransomwaretracker.abuse.ch/blocklist/
1) Domain:
https://ransomwaretracker.abuse.ch/downloads/RW
_DOMBL.txt
2) ignoring regex :
(^#|^\s*$)
3) fields :
domain:$1,description:Ransomware_domain
13
6
© 2017 SPLUNK INC.
Add Ransomware Threat Intelligence

detect active ransomware in the browser or other applications
1) Ransomware ServerIPs:
https://ransomwaretracker.abuse.ch/downl
oads/RW_IPBL.txt
2) ignoring regex :
(^#|^\s*$)
3) fields :
ip:$1,description:Ransomware_IP
13
7
© 2017 SPLUNK INC.
End Point Analysis

© 2017 SPLUNK INC.
Know Your Endpoint:

Microsoft Sysmon Primer
▶ TA Available on the App Store

• https://splunkbase.splunk.com/app/1914
▶ Great blog post to get you started

Blog Post:
http://blogs.splunk.com/2014/11/24/monitoring-network-traffic-with-sysmon-and-splunk/
Sysmon (System Monitor) is part of the Windows
Sysinternals Suite and can be downloaded for free.
It is a system service and device driver, that logs ▶ Increases the fidelity of Microsoft Logging
system activity to the EventLog.
• e.g. Process Execution, CommandLine Logging
© 2017 SPLUNK INC.
Hands-on Exercise
Question 12
▶ What was the most likely IP address of we8105desk on 24AUG2016?
Answer : 192.168.250.100
© 2017 SPLUNK INC.
Host Centric Log Source

© 2017 SPLUNK INC.
Vast Majority of Source IPs from Single Address

© 2017 SPLUNK INC.
src v src_ip v src_host

© 2017 SPLUNK INC.
Another way to find the answer : -
▶ What was the most likely IP address of we8105desk on 24AUG2016?

• 192.168.250.100
© 2017 SPLUNK INC.
Ransomware Picture
Hostname: we8105desk
IP: 192.168.250.100
© 2017 SPLUNK INC.
Process Execution
sourcetype=x* | stats values(ParentImage) by process
© 2017 SPLUNK INC.
CommandLine
sourcetype=x* CommandLine=* | stats values(CommandLine) by Computer,process
© 2017 SPLUNK INC.
Powershell Process
sourcetype=x* process=*powershell.exe | stats values(CommandLine) by Computer,process
© 2017 SPLUNK INC.
Be careful of this !!
sourcetype=x* process=*powershell.exe CommandLine=”*-enc*” | top
CommandLine
© 2017 SPLUNK INC.
Hands-on Exercise
Question 13
▶ During the initial Cerber infection a VB script is run. The entire script from this
execution, pre-pended by the name of the launching .exe, can be found in a field
in Splunk. What is the length in characters of this field ?
Answer : 4490
Very long CommandLine ?
© 2017 SPLUNK INC.
sourcetype="XmlWinEventLog:Microsoft-Windows-
Sysmon/Operational" | eval len=len(CommandLine) | table
User, len, CommandLine | sort - len
sourcetype="XmlWinEventLog:Microsoft-Windows-Sysmon/Operational" |
eval len=len(ParentCommandLine) | table User, len, ParentCommandLine |
sort - len
15
1
© 2017 SPLUNK INC.
What Do We Have?
▶ What data set will show execution of commands and the specific commands
executed?
• Sysmon
▶ Do we know what kind of a file we are looking for?
• .exe
▶ Let’s start there
© 2017 SPLUNK INC.
Initial Search
© 2017 SPLUNK INC.
Refined Search
Something Doesn’t Seem Right
© 2017 SPLUNK INC.
With Our Tweaks

© 2017 SPLUNK INC.
The very long script is indeed the new payload
▶ During the initial Cerber infection a VB script is run. The entire script from this
execution, pre-pended by the name of the launching .exe, can be found in a field
in Splunk. What is the length in characters of this field?
• 4490
© 2017 SPLUNK INC.
Hands-on Exercise
Question 14
▶ Bob Smith's workstation (we8105desk) was connected to a file server during the
ransomware outbreak. What is the IP address of the file server?
Answer : 192.168.250.20
© 2017 SPLUNK INC.
Where is our data?
▶ What data could be used to find shares or server connections?

• Registry or sysmon
▶ What might I expect to see in a ransomware outbreak if I was connected to a
share?
• Ransomware often will encrypt data not only on the local drive but on any connected file shares
• Lots of connections = lots of log events
• Look for many connections from Bob’s workstation to other internal IPs
▶ Do we know Bob’s system name?
• Yes, both by IP and hostname
© 2017 SPLUNK INC.
Identifying the src of sysmon Events

© 2017 SPLUNK INC.
sysmon Event Description

© 2017 SPLUNK INC.
Back to Our Old Friend, the stats command

© 2017 SPLUNK INC.
Registry sourcetype
© 2017 SPLUNK INC.
fileshare
▶ Bob Smith's workstation (we8105desk) was connected to a file server during the
ransomware outbreak. What is the IP address of the file server?
• 192.168.250.20
© 2017 SPLUNK INC.
Hands-on Exercise
Question 15
▶ How many distinct PDFs did the ransomware encrypt on the remote file server ?
Answer : 257
© 2017 SPLUNK INC.
Question 207
Intermediate
▶ How many distinct PDFs did the ransomware encrypt on the remote file server?
▶ Search the sourcetype WinEventLog:Security for the remote fileserver. Narrow

your search by looking for .pdf files.
© 2017 SPLUNK INC.
Hmmm...
© 2017 SPLUNK INC.
Taking a Step Back

© 2017 SPLUNK INC.
Searching with the Correct Destination

© 2017 SPLUNK INC.
Ransomware Picture
Encryption
cerberhhyed5frqa.xmfir0.win
Initial Connection
solidaritedeproximite.org
Hostname: we8105desk
IP: 192.168.250.100
USB Key Name: MIRANDA_PRI File Server
VB script executed – Excessive Length
Hostname: we9041srv
IP: 192.168.250.20
© 2017 SPLUNK INC.
Distinct Count of Relative_Target_Name

© 2017 SPLUNK INC.
Use dc() to find the final answer

Intermediate
▶ How many distinct PDFs did the ransomware encrypt on the remote file server?
• 257
© 2017 SPLUNK INC.
Hands-on Exercise
Question 16
▶ What is the first parent process ID of 121214.tmp ?
Answer : 3968
© 2017 SPLUNK INC.
CommandLine
© 2017 SPLUNK INC.
Question 208
Beginner
▶ What is the parent process ID of 121214.tmp?
• 3968
© 2017 SPLUNK INC.
Detect Known APT

© 2017 SPLUNK INC.
Lessons learned from the “SWIFT” Attack
Source: BAE SYSTEMS
17
6
© 2017 SPLUNK INC.
IOC for detection

1) In your Network Traffic / Firewall Data / Proxy:Command and Control
IP Address Geo Location

196.202.103.174 Egypt
2) In your Endpoint Data / Sysmon / Carbon Black / THOR Scanner:

SHA1 Compile time Size (bytes) Filename
525a8e3ae4e3df8c9c61f2a49e38541d196e9228 2016-02-05 11:46:20 65,536 evtdiag.exe
76bab478dcc70f979ce62cd306e9ba50ee84e37e 2016-02-04 13:45:39 16,384 evtsys.exe
70bf16597e375ad691f2c1efa194dbe7f60e4eeb 2016-02-05 08:55:19 24,576 nroff_b.exe
6207b92842b28a438330a2bf0ee8dcab7ef0a163 N/A 33,848 gpca.dat
3) Processes which have never been seen before such as new network communication on unknown ports or to a
rare IP-address could be an early indicator of an attack and need to be reviewed
17
7
© 2017 SPLUNK INC.
Bad Rabbit
1. Block the execution of files

c:\windows\infpub.dat and
c:\Windows\cscc.dat.
(infpub.dat is a valid Windows

file, but lives in
c:\Windows\System32\DriverSt
ore, not c:\Windows)
1. Disable WMI service (if it’s

possible in your
https://www.cybereason.com/blog/cybereason- environment) to prevent the
researcher-discovers-vaccine-for-badrabbit- malware from spreading
ransomwar 17 over your network.
8
© 2017 SPLUNK INC.
IOC for detection
For more info, read this :

https://www.welivesecurity.com/2
017/10/24/bad-rabbit-not-petya-
back/
17
9
© 2017 SPLUNK INC.
Hunting Unknown
Threat
Inside Threat : How to detect and trace Unknown Data
© 2017 SPLUNK INC.
Exfiltration ??
Same events can have different security meanings, based on sequence:
Event 1 … 13:01:21
Event 2 … 13:42:17
Action 3
Known DNS Unknown DNS
= Misconfiguration
Note “Windows
event”
Analyst /
Investigator
What happened? Unknown DNS Unknown DNS
= Recon, Lateral Movement
= Exfiltration
If event 1, then
event 2, then… Unknown DNS C&C Site Lots of DNS Event
Ah – ha, that’s
how they got in.
Now what infected
the host?
18
1
DNS
© 2017 SPLUNK INC.
Tunneling
18
182
2
© 2017 SPLUNK INC.
DNS
Exfiltration
18
3
© 2017 SPLUNK INC.
Case Study 2 : DNS Exfiltration Indicators (IOC)

• Look @ all DNS traffic for multiple levels of DNS strings. look for hexadecimal strings.
• Look for this 3rd level to be less than 40 bytes in length... like *.domain.com, where * is longer
than 40 bytes
• Look for multiple DNS Name lookups to sketchy foreign domains, and look at the frequency in a
short time span.
• DNS TXT or SRV record queries to any of these suspicious domains.
• ANY DNS response to a loopback or RFC 1918 space. (5.0.0.0/8, 10.0.0.0/8, 192.168.0.0/16,
172.16.0.0/12) indicates a C2 channel
• Multiple DNS queries to the same non-obvious or foreign domain during off-hours times in the
office = check for frequency, and periodicity.
• DNS queries to dynamic DNS providers (like OpenDNS)
• DNS queries not followed by a proxy request for connection
• Recurring interval or beacon following any of the above (zero variance behavior)
• DNS requests or returns that are base64 encoded
18
4
© 2017 SPLUNK INC.
Hands-on Exercise
Question 17
▶ The internal IP address ranges are 10.0.0.0/8, 172.16.0.0/12, 192.168.0.0/16.

Finding which unauthorized DNS servers has the most number of internal user
connecting to ?
Answer : 8.8.8.8
Finding unauthorized DNS servers
© 2017 SPLUNK INC.
• DNS Tunneling
• DNS Spoofing
Host Subnet
index=stream sourcetype=stream:dns dest_port=53 dest_ip!=10.0.0.0/8 | stats count by

dest_ip
index=bro sourcetype=bro_dns dest_port=53 dest_ip!=10.0.0.0/8 | stats count by dest_ip
tag=dns dest_port=53 dest_ip!=10.0.0.0/8 | stats count by dest_ip
18
6
Finding unauthorized DNS servers
© 2017 SPLUNK INC.
18
7
sourcetype=bro_dns dest_port=53 dest_ip!=10.0.0.0/8 dest_ip!=192.168.0.0/16 dest_ip!=172.16.0.0/16
© 2017 SPLUNK INC.
| stats dc(src_ip) values(src_ip) by dest_ip

Finding clients connecting to multiple DNS servers
© 2017 SPLUNK INC.
• DNS Spoofing
• DNS Exfil
• DNS Tunneling
tag=dns dest_port=53 dest_ip!=10.0.0.0/8 |bucket _time span=1s | stats

VALUES(dest_ip) AS IP_List dc(dest_ip) AS distinct by _time src_ip| search distinct > 2|
table src_ip IP_List distinct
tag=dns dest_port=53 | stats values(dest_ip) as destiplist dc(dest_ip) as dc by

src_ip | where dc > 2
18
9
Finding clients connecting to multiple DNS servers
© 2017 SPLUNK INC.
19
0
© 2017 SPLUNK INC.
Hands-on Exercise
Question 18
▶ There is data exfiltration by using DNS tunneling? Find the IP addresses of the
source PC.
Answer : 10.143.15.56, 10.144.15.68, 10.153.15.6

Finding Clients with extremely Looooooooooooong
© 2017 SPLUNK INC.
• DNS Tunneling
queries
• DNS Exfil
Find anything that is 2 standard deviations
sourcetype=bro_dns |eval len=len(query) | eventstats stdev(len) AS stdev avg(len) AS

avg p50(len) AS p50 | eval length=len(query) | where length>(stdev*2) | stats count by
length stdev avg p50 qtype_name query | sort -length
Finding queries over 200 characters long
sourcetype=bro_dns | eval length=len(query) |search length>200| stats count by query
19
2
Finding Queries Over 200 characters
© 2017 SPLUNK INC.
19
3
Bonus Exercise
© 2017 SPLUNK INC.
• Check which IP pairs ( source IP & dest IP ) has sent out longest
domain name query
sourcetype=bro_dns | eval ippair = src_ip."-".dest_ip | eval query_len =

len(query) | stats values(ippair) count by query_len
19
4
Finding Queries Two Standard Deviations Over Normal
© 2017 SPLUNK INC.
19
5
© 2017 SPLUNK INC.
Shannon Entropy to detect DGA Domains
Layman’s definition: a score reflecting the randomness or

measure of uncertainty of a string
▶ Examples
• The domain aaaaa.com has a Shannon Entropy score of 1.8 (very low)
• The domain google.com has a Shannon Entropy score of 2.6 (rather low)
• A00wlkj—(-a.aslkn-C.a.2.sk.esasdfasf1111)-890209uC.4.com has a Shannon
Entropy score of 3 (rather high)
© 2017 SPLUNK INC.
instead of looking for domains with high entropy, we are

looking for SUB domains with high entropy.
© 2017 SPLUNK INC.
Detecting Data Exfiltration
▶ TIPS
• Leverage our Bro DNS data index=bro sourcetype=bro_dns
• Calculate Shannon Entropy scores | `ut_parse(query)`
• Calculate subdomain length | `ut_shannon(ut_subdomain)`
• Display details | eval sublen =
length(ut_subdomain)
| table ut_domain ut_subdomain
ut_shannon sublen
© 2017 SPLUNK INC.
▶ TIPS … | stats
• Leverage our Bro DNS data count
• Calculate Shannon Entropy scores avg(ut_shannon) as avg_sha
• Calculate subdomain length avg(sublen) as avg_sublen
• Display count, scores, lengths, deviations stdev(sublen) as stdev_sublen
by ut_domain
| search avg_sha>3
avg_sublen>20 stdev_sublen<2
© 2017 SPLUNK INC.

▶ RESULTS
• Exfiltrating data requires many DNS requests – look for high counts
• DNS exfiltration to mooo.com and chickenkiller.com

© 2017 SPLUNK INC.
Summary: DNS exfiltration
▶ Exfiltration by DNS and ICMP is a very common technique

▶ Many organizations do not analyze DNS activity – do not be like them!
▶ No DNS logs? No Splunk Stream? Look at FW byte counts
© 2017 SPLUNK INC.
Latest APT
Operations
© 2017 SPLUNK INC.
Good Bedtime reading websites

APT cases and lists IOT malware Checking
– https://www.shadowserver.org/
– https://www.threatminer.org/
– https://www.shodan.io
– https://www.shodan.io/search?
Specific Investigation & Research query=country%3A%22hk%22
– https://www.threatconnect.com/blog +port%3A2323
– https://www.cybereason.com/blog
IP/Domain/IOC Reputation
Checking
– http://www.robtex.com
– https://www.threatminer.org/
– https://www.shodan.io
© 2017 SPLUNK INC.
Shodan report active mirai victims

© 2017 SPLUNK INC.
Case Study :
Operation Cobalt Kitty: A large-scale APT in Asia carried out by the OceanLotus Group (also
known as, APT-C-00, SeaLotus and APT32)
https://www.cybereason.com/labs-operation-cobalt-kitty-a-large-scale-apt-in-asia-carried-out-by-the-oceanlotus-group/
20
5
© 2017 SPLUNK INC.
Phase 1 : Fileless operation (PowerShell and Cobalt Strike

payloads)
Scheduled task 1:
Downloads a COM scriptlet that
redirects to Cobalt Strike
payload
20
6
© 2017 SPLUNK INC.
Phase 1 : Fileless operation (PowerShell and Cobalt Strike

payloads)
Scheduled task 2:
Uses Javascript to download
a Cobalt Strike Beacon
20
7
© 2017 SPLUNK INC.
Fileless payload delivery infrastructure
20
8
© 2017 SPLUNK INC.
Phase 2 : Backdoors exploiting DLL-hijacking and using

DNS tunneling
•Windows Search (vulnerable applications: searchindexer.exe
/searchprotoclhost.exe)
•Fake DLL: msfte.dll (638b7b0536217c8923e856f4138d9caff7eb309d)
20
9
© 2017 SPLUNK INC.
Phase 2 : Backdoors exploiting DLL-hijacking and using

DNS tunneling
DNS Tunneling as C2 channel
21
0
© 2017 SPLUNK INC.
“8.8.8.8” – Google’s DNS server

– the malicious domain is “hiding” inside the DNS packet:
21
1
© 2017 SPLUNK INC.
Phase 3 : Novel MS Outlook backdoor and lateral

movement spree
Outlook macro backdoor
21
2
© 2017 SPLUNK INC.
Phase 4 : New arsenal and attempt to restore PowerShell

infrastructure
e.g. Modified NetCat
After a four week lull and no apparent
malicious activity, the attackers returned to the
scene and introduced new and improved tools
aimed at bypassing the security mitigations
that were implemented by the company’s IT
team.
These tools and methods mainly allowed

them to bypass the PowerShell execution
restrictions and password dumping
mitigations
21
3
© 2017 SPLUNK INC.
What’s Next?
Experienced
Anatomy of An Attack (Future)
Boss of the SOC v2
Investigating with Splunk

(Boss of the SOC v1 Data)
ES BOTS Hands-On
Threat Hunting Essentials -

Search for Security
Introductory
© 2017 SPLUNK INC.
Splunk Security Essentials

https://splunkbase.splunk.com/app/3435/
Learn to improve your security using

Splunk’s analytics-driven security:
✓ Common use cases and examples to get started
✓ Data onboarding guides for top data sources
✓ Understand how to improve your security
✓ Scales from small to massive companies
✓ Save searches, send results to ES/UBA
You can learn to solve use cases today

for free, then use with Splunk solutions
© 2017 SPLUNK INC.
Splunk Security Essentials Covered Use Cases

© 2017 SPLUNK INC.
Data Onboarding Guides
▶ AWS CloudTrail + VPC Flow

▶ Cisco ASA
▶ Linux Security Logs
▶ Microsoft Sysmon
▶ Microsoft Office 365
▶ Palo Alto Networks
▶ Stream DNS
▶ Symantec AV
▶ Windows Security
© 2017 SPLUNK INC.
Does SSE + Splunk Enterprise = SIEM?

Enterprise + SSE ES: All of that, plus…
▶ Prescriptive Security Content ▶ Additional Prescriptive Content (ESCU)
▶ Search ▶ Integrations
▶ Visualizations ▶ Contextual Data (Threat, Vulnerability,
▶ Alerting Asset, Identity)
▶ Real-time & Batch Analytics ▶ Analyst Interface/Workflow

▶ Incident/Case Management
▶ Dashboards
▶ Lots of canned reports
▶ Decision Support
GAP! ▶ Risk Scoring
▶ Data Sharing
▶ Light Automation (Adaptive Response)
© 2017 SPLUNK INC.
Next Steps
▶ If you would like to take the data set home with you and explore further, now you
can!
• https://www.splunk.com/files/downloads/botsv1_data_set.tgz
• Contains a README and an app with indexes of the data
▶ Read our blog series called Hunting with Splunk!
• https://www.splunk.com/blog/2017/07/06/hunting-with-splunk-the-basics.html
• Ever increasing, covers many of the topics we talked about today
© 2017 SPLUNK INC.
Helpful Links
▶ Splunk Quick Reference: https://www.splunk.com/pdfs/solution-guides/splunk-

quick-reference-guide.pdf
▶ Splunk Search Reference:

http://docs.splunk.com/Documentation/Splunk/latest/SearchReference/WhatsInT
hisManual
▶ Hunting with Splunk Blog Series:

https://www.splunk.com/blog/2017/07/06/hunting-with-splunk-the-basics.html
▶ Geomapping Blog: https://www.splunk.com/blog/2017/04/25/mapping-with-

splunk.html
© 2017 SPLUNK INC.
Thank You

Splunk 201 - Threat Hunting Workshop

Uploaded by

Document Information

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Splunk 201 - Threat Hunting Workshop

Uploaded by

Copyright:

Available Formats

© 2017 SPLUNK INC.

Jun 2018 | Version 1.1

How to access the Lab

– User : user01 .. user30

How to access the Lab

– User : user01 .. user30

20+ Years as SOC Advisor

Organization of the SOC/CSIRT Team

Threat Hunting Techniques

Gain Access Create additional Conduct

Host What created the How was

Auth - User Roles

Data Source Mapping

Web Email Endpoint Proxy/DNS

CMDB and Threat Intelligence

Splunk Security Portfolio

DATA PLATFORM ANALYTICS OPERATIONS

> Universal indexing ES CONTENT

Machine Learning Toolkit

Splunk Security Portfolio

Search All Your Machine Data

▶ Real-time and historical data Search all

▶ Over 140 commands including

Auto-detected fields and values

Extract Fields Anytime

▶ Simple field extraction

Splunk Commands Used

Ex 1 : Using Splunk for Security

Luckily Everything Is Captured In Splunk

Reviewing All The Data Available

▶ What is the likely IP address of someone scanning imreallynotbatman.com for

Selected and Interesting Fields

Default Selected Fields

Fast v Smart v Verbose Mode

Fast Mode with Statistics

Implicit AND in Search

What is the likely IP address scanning

Adding Transformational Search Commands

▶ Search commands are separated by |

Building on Our Previous Search

Introducing the stats command

▶ What is the IP address of the server hosting imreallynotbatman.com?

Finding the Server IP

▶ What IP address is likely attempting a brute force password attack against

HTTP Get v Post Traffic Flow

▶ HTTP GET will generally retrieve data

▶ Wildcards allow us to only have a portion of a value and still search on it

▶ What IP address is likely attempting a brute force password attack against

stats command for the win...

▶ What company created the web vulnerability scanner used by Po1s0n1vy?

Check the HTTP header payload

You can find this from the stream:http

You can also found the similar information from

What If You Don’t Know What Acunetix Is?

▶ If you see something in the logs

At the halfway mark...

▶ What was the first brute force password used?

What was the first brute force password used?

Newest at the Top

Add reverse Command

Extracting the Password from the String

▶ The search command is implied at the start of every search

head & tail Commands

▶ Opposite of head is tail

sourcetype=weblog .apache.struts2 wget OR “service iptables” OR chmod