Professional Documents
Culture Documents
gh
Threat Hunting: This is the Way
Ri
ll
Fu
ns
GIAC (GCIH) Gold Certification
ai
Author: Melissa Raney, melissa.c.raney@gmail.com
et
Advisor: Bryan Simon
rR
ho
Accepted: 9-October-2021
ut
,A
te Abstract
itu
Threat hunting is an advanced defensive security discipline that is usually conducted by the only
most skilled members of a SOC team – if at all. Many organizations are still not conducting any
st
threat hunting for various reasons. This research is aimed at those organizations. By delivering an
In
easy-to-implement, no-cost threat hunting program using only existing SOC resources, this
project seeks to remove any remaining obstacles - or excuses – for organizations that don’t
NS
gh
1. Introduction
Ri
ll
Threat hunting has gained traction as a defensive security practice in recent years,
Fu
but oddly, the cyber security community still remains conflicted on a solid definition for
ns
it. Therefore, it is no surprise there are so many misconceptions about the practice and its
ai
potential value. The SANS 2020 Threat Hunting Survey revealed that only half of
et
individuals saw value in threat hunting, while another 30% acknowledged they did not
rR
even know how to begin (Fuchs & Lemon, 2020). These survey results reveal a
ho
significant missed opportunity that this project aims to remedy by demonstrating both
ut
concrete and intangible benefits of threat hunting – and guidance for how to get started.
,A
Some in the community who have not been convinced of the value of threat
te
hunting can be attributed to one of many prevailing misconceptions, including the notion
itu
that all threat hunting is highly sophisticated or requires a lot of expertise and
st
complicated tools. As Michael Collins, author of the 2018 O'Reilly e-book Threat
In
Hunting, depicts it, threat hunting "is highly… self-directed and carried out by senior
NS
analysts. Good threat hunters are investigators, developers, teachers, and highly
SA
autodidactic” (Collins, 2018). Given these heroic sounding qualities, it is little wonder
full-time threat hunters are rare on security teams (Fuchs & Lemon, 2020). Further, with
e
Th
typical salaries on par with their elite skills, the perceived cost of these individuals may
lead some Security Operations Center (SOC) managers to think that the entire threat
22
Collins goes on to advise managers to "cultivate threat hunters from junior staff"
©
and focus on skill development over an analyst's familiarity with specific tools (2018).
This recommendation has two core benefits: not only is threat hunting demonstrably
valuable to an organization's security posture, but the methodology itself also has a
compelling bonus. Threat hunting activities can help develop, motivate, and retain staff,
which makes it one of the top goals SOC managers should set for themselves right now.
1.1. Thesis
This project aims to prescribe the minimal people, process, and technology
required to launch a low-to-no-cost threat hunting program within most existing Security
© 2022
Internal UseThe SANS Institute
- Confidential Author retains full rights.
ts
Threat Hunting: This is the Way 3
gh
Operations Centers (SOCs). Whether the SOC has a team of eight or 80 analysts, this
Ri
project will provide the framework for a readily consumable, easy-to-implement
ll
Fu
program, leveraging only the organization's existing resources. The threat hunting
activities prescribed by this program will deliver increased visibility into the security of
ns
the enterprise and improve the overall security posture. Still, there is an even more
ai
compelling reason for organizations to adopt it: threat hunting inherently provides
et
rR
continuous challenge, training, and growth opportunities for SOC analysts, and those
intrinsic experiences ultimately improve job satisfaction and employee retention (Steil, de
ho
Cuffa, Iwaya, & Pacheco, 2020).
ut
,A
1.2. Threat Hunting as a Disciplinete
While day-to-day SOC operations focus on reacting to alerts and establishing
itu
whether an incident has occurred, threat hunting is fundamentally proactive, looking for
st
proof that an undetected attacker is already in the network. As such, threat hunting
In
activities do not start with an alert in the system but rather a theory about what it would
NS
look like if a given attack had been successful. In describing the threat hunting work he
does in both the public and private sector, Mike Mallon explained he always “assumes
SA
the worst” (Mallon, 2021), which not only removes any mindset that a given occurrence
e
could not possibly have happened, the threat hunting mindset assumes that it has in fact
Th
happened and that evidence can be found. All professional threat hunters interviewed for
22
this project agreed on this idea – and the following: “take nothing for granted.” James
20
Pope and Neil Wyler, both former principal threat hunters at RSA, relayed stories of
©
The output of a hunt adds insight to the organization either way. Expected
findings, like verifying RDP traffic is or is not present on the network in accordance with
organizational policy, provides environmental context and critical situational awareness
© 2022
Internal UseThe SANS Institute
- Confidential Author retains full rights.
ts
Threat Hunting: This is the Way 4
gh
to the organization. Unexpected findings, like looking for RDP traffic but stumbling upon
Ri
a completely unrelated violation, provides an opportunity to improve organizational and
ll
Fu
network hygiene.
ns
1.3. What Is (and Isn’t) Threat Hunting
ai
Threat hunting is hardly a new concept, but as of 2021, there is still no consensus
et
on so much as a definition for it, much less an agreement on what a full program should
rR
look like. Authors of the SANS 2020 Survey on Threat Hunting advised that one of "the
ho
most crucial topics that must be addressed… (is) … establishing a common
ut
understanding of threat hunting” (Fuchs & Lemon, 2020). For clarity and perhaps
,A
someday, consensus, it is described here as follows: “Threat hunting is a proactive, data-
te
driven methodology that applies an iterative search for specific events within enterprise
itu
data, going beyond standard detection capabilities – a process that is then repeated in
st
recurring cycles over time” (Gao, et al., 2021). Although these searches may employ
In
pieces of underlying automation, threat hunting is a largely manual process. This process
NS
constitute the bulk of day-to-day SOC operations. That work provides the critical
20
foundation for defensive operations but does not constitute "threat hunting," as even valid
©
alerts are typically representative of lower-level attacks. As authors of the recent journal
article “Enabling Efficient Cyber Threat Hunting With Cyber Threat Intelligence”
warned, SOC defenses relying solely "on these low-level, fragmented indicators can be
easily evaded when the attacker re-purposes the tools and changes their signatures” (Gao,
et al., 2021).
© 2022
Internal UseThe SANS Institute
- Confidential Author retains full rights.
ts
Threat Hunting: This is the Way 5
gh
(detection) is insufficient: "Proactively seeking unknown, malicious behaviours and
Ri
looking for anomalies inside the network is the right approach… (because) today's cyber
ll
Fu
attackers do not follow any specific, predefined rules or paths of engagement" (Bhardwaj
& Goundar, 2019).
ns
ai
et
1.4. The Case for Threat Hunting
rR
For organizations not yet sold on the value of threat hunting, there are many good
ho
reasons to reconsider it. First and foremost, detection alone has proven ineffective for
ut
identifying many types of cyber threats, particularly advanced, persistent, or staged
,A
attacks (Akinrolabu, Agrafiotis, & Erola, 2018). Attackers and their techniques continue
te
to change and develop rapidly, so staying ahead of the more advanced threats requires
itu
similar levels of vigilance, creativity, and determination on the part of the SOC (Tatam,
st
Shanmugam, Azam, & Kannoorpatti, 2021). Second, there are tangible gains to be had in
In
SANS survey reported massive improvements due to adopting threat hunting practices in
SA
their organizations; these benefits included fewer breaches, reduced attack surface
exposure, and improved speed and accuracy of SOC responses (Bhardwaj & Goundar,
e
Th
2019).
22
© 2022
Internal UseThe SANS Institute
- Confidential Author retains full rights.
ts
Threat Hunting: This is the Way 6
gh
Still, the most compelling catalyst for this project was using threat hunting as a
Ri
mechanism to help retain SOC staff. This proposed program provides specific, tangible
ll
Fu
results for improving an organization's overall security posture – but also its SOC staff.
The threat hunting tasks prescribed here provide a recurring opportunity for staff training
ns
and development, which has been demonstrated to increase job satisfaction and,
ai
ultimately, employee retention (Kazmierczyk, Romashkina, & Macholak, 2020). Further,
et
rR
retaining staff is a nearly universal challenge for most SOC managers today. The SANS
Survey “Closing the Critical Skills Gap for Modern and Effective Security Operations
ho
Centers (SOCs)” reported that 77% of respondents said retaining security staff was "a
ut
problem" or at least "somewhat difficult“ (Filkins & Pescatore, July 2020). As such, the
,A
inherent benefits of this kind of program are vast.
te
itu
widely acknowledged correlation between training and employee retention may be even
NS
stronger among those of us working in technology (Steil, de Cuffa, Iwaya, & Pacheco,
2020). A 2020 study on the link between learning opportunities and employee retention
SA
and technicians presented significant positive correlations to stay and significant negative
Th
correlations to leave the organization (Steil, de Cuffa, Iwaya, & Pacheco, 2020)." This
22
research urges SOC managers to place higher priority on training and development for
20
Finally, threat hunting can also highlight gaps in an organization's visibility of the
digital enterprise, helping to justify capital expenditures for new security systems and
tools. As any security manager knows, to the ability to justify the team's function and
need for tools is priceless.
2. Program Framework
This program will outline the people, process, and technology needed to develop a
foundational threat hunting program that most any size SOC organization can consume.
Threat hunting tasks will be assigned to, and ultimately conducted by, each tier of the
© 2022
Internal UseThe SANS Institute
- Confidential Author retains full rights.
ts
Threat Hunting: This is the Way 7
gh
SOC's existing staff. Tasks assigned will be commensurate with the skills and experience
Ri
typical for those positions. SOC members will use their organization's existing tools to
ll
Fu
conduct the tasks – typically queries against a log and network data database. Finally, this
program will offer an assortment of metrics and report suggestions to help justify,
ns
measure, and communicate the output of the SOC's threat hunting efforts and ultimately
ai
support internal marketing efforts.
et
rR
One of the program's key components is a recurring and compulsory cadence to
ho
focus exclusively on threat hunting tasks. SOC management should allocate a consistent
ut
schedule for threat hunting for all SOC staff members. Short but consistent intervals,
,A
such as an hour per workday or two hours a couple of times a week, are optimal. This
time must be reserved exclusively for threat hunting activities without any expectations
te
itu
for team members to maintain their primary functions during that time. Trying to do both
simultaneously will only frustrate staff and ensure that neither is done correctly, so SOC
st
In
managers should plan to fully relieve staff of their primary responsibilities during their
NS
While threat hunting tasks do not always lead to the vulnerabilities it set out to examine,
Th
it frequently uncovers other findings that are just as valuable and provides opportunities
22
to work with other teams. Neil Wyler, who is also known by his hacker handle "Grifter,"
20
stated that he has also seen benefits to the broader IT organizations as a result of threat
©
hunting (Wyler, 2021). By connecting the SOC with other internal teams while
investigating and remediating findings, other departments are given more insight and a
sense of ownership into security issues. Given that remediation actions are frequently
handled by different groups like server administrators or network administrators, this
kind of cross-functional engagement can forge new productive working relationships and
give the IT organization more investment into security.
2.1. People
Disavowing the idea that threat hunting is too difficult or too complex for any
individual or SOC team, this program's goal is to make everyone a threat hunter. Instead
© 2022
Internal UseThe SANS Institute
- Confidential Author retains full rights.
ts
Threat Hunting: This is the Way 8
gh
of isolating those tasks to one or two individuals (or being foregone entirely), this
Ri
program aims to train everyone in the SOC to be a threat hunter. This approach
ll
Fu
distributes the workload to negligible levels for each individual while also creating new
learning opportunities for security analysts at every level in the organization. By
ns
spreading out the tasks classically handled by a dedicated threat hunter, existing SOC
ai
resources of all levels can amass new threat hunting skills over time and start adding
et
rR
value to their organizations straightaway.
ho
At the 2020 RSA Conference, a presentation from Information Systems Security
ut
Association (ISSA) revealed that 70% of organizations in their survey reported being
,A
"impacted by the cybersecurity skills shortage," citing increased workloads on existing
staff and lingering open requisitions (Oltsik & Alexander, 2020). SOC managers who
te
itu
relate to this metric may be reticent to take on more work, but this small investment of
time into each team member directly supports SOC efficiency and productivity efforts, as
st
In
levels of maturity. However, the typical SOC structure employs a tiered level of expertise
and support for handling an enterprise's day-to-day defensive data security needs.
22
Traditional SOC functions include monitoring and responding to alerts and triage of
20
security-related incidents (Kokulu et al., 2019). SOC teams are frequently also
©
Tier one analysts serve as the front line of the defensive security organization,
monitoring for and responding to alerts in addition to conducting the initial investigation
into potential security incidents (Kokulu et al., 2019). Tier two analysts perform
additional analysis to pursue incident response and resolution, typically escalated from
tier one, and may also take various response actions such as blocking ports (Kokulu et al.,
2019). Finally, tier three analysts represent the highest level of skills and subsequently
perform more in-depth analysis and, in some cases, also threat hunting.
© 2022
Internal UseThe SANS Institute
- Confidential Author retains full rights.
ts
Threat Hunting: This is the Way 9
gh
It is worth noting that, while their structures are usually similar, there are both in-
Ri
house and outsourced versions of SOC operations. In-house SOC organizations are
ll
Fu
comprised of individuals who are employed by the same enterprise they defend, whereas
outsourced SOCs provide their function as a service to a given enterprise (Kokulu et al.,
ns
2019). Many elements of this proposed program could apply to either type but are
ai
primarily intended for the internal SOC.
et
rR
2.1.2. Separation of Duties
ho
Separation of duties is a fundamental security principle keenly relevant to threat
ut
hunting. This basic premise segregates the privileges for managing and controlling a
,A
given activity so that no single entity in the process can manipulate it to its advantage
te
(Baykara, 2021). The separation of duties between the SOC's day-to-day monitoring-
itu
driven activities and threat hunting can also lead to better insights, improving the SOC
st
analysts' primary jobs. This concept was illustrated by Threat Hunter, James Pope, who
In
related a story about his early career days overseeing movie theater management. James
NS
explained that when he had a manager at one store who struggled to identify and correct
specific fundamental issues, he would take them into an identical theater in another
SA
neighborhood where they would inevitably see those overlooked findings (Pope, 2021)!
e
Taking individuals out of their regular day-to-day routine can help provide new insights
Th
and a fresh perspective, whether it’s movie theater management or SOC operations.
22
surveys. A SANS 2019 SOC survey revealed that 58% of SOC managers reported a lack
of skilled staff (Crowley, 2019). More recently, some SOC managers are also facing
increased budget constraints, including hiring freezes and other cost-cutting measures
(Filkins & Pescatore, July 2020), which are possibly associated with COVID-19
pandemic economic repercussions. Less than half of respondents in the survey had any
organizational metric for hiring a new SOC staff member (Filkins & Pescatore, July
2020), which likely contributes to the skills shortage many managers face since that
leaves them with only attrition-based replacement hiring.
© 2022
Internal UseThe SANS Institute
- Confidential Author retains full rights.
ts
Threat Hunting: This is the Way 10
gh
One of the key "avenues for improvement" identified in the SANS SOC survey
Ri
was "training staff by providing opportunities to learn and develop” (Filkins & Pescatore,
ll
Fu
July 2020), and this program aims to support that goal. Further, interview respondents
who had found success in improving the effectiveness and efficiency of their SOC
ns
operations did so by focusing on increasing staff skills in key areas” (Crowley, 2019)
ai
which provides further evidence for the importance of novel learning opportunities for
et
rR
analysts, like those proposed in this program.
ho
The struggle to develop meaningful performance metrics is another major
ut
challenge for many SOCs. As researchers at Arizona State University observed, this void
,A
also impacts the broader security community (Kokulu, et al., 2019). The authors stated:
"Current quantitative metrics, such as the number of incidents and average response time,
te
itu
are not effective in measuring SOC success because each security event has unique
severity and consequences (Kokulu, et al., 2019).” The researchers argued that this lack
st
In
of insight into the issues modern SOCs are facing leaves the academic community ill-
NS
prepared to address those woes. Again, this program's output may help SOC managers
improve the quality of their reporting by providing new insights into the security posture
SA
2.2. Process
20
© 2022
Internal UseThe SANS Institute
- Confidential Author retains full rights.
ts
Threat Hunting: This is the Way 11
gh
activities far enough apart to take undue time to resume the tasks and ultimately regain
Ri
lost momentum. Whatever the schedule, the SOC manager should create an equitable,
ll
Fu
consistent opportunity for everyone on the team. Managers will also want to meet
periodically as a threat hunting team to share stories and experiences.
ns
ai
2.2.2. Threat Intelligence – MITRE ATT&CK
et
Successful threat hunting requires a certain level of insight into the tools,
rR
techniques, and procedures leveraged by cyber attackers (Daszczyszak, Ellis, Luke, &
ho
Whitely, 2019).This too can be perceived as a barrier to entry to threat hunting for some
ut
organizations – but should not be. Countless sources of intelligence about attacker
,A
activities exist today and can be consumed in a variety of ways. Intelligence data is
te
available in structured feeds like those provided by Structured Threat Information
itu
and Analysis Centers (ISAC). SOC managers may also consider paid and free
NS
subscriptions, open-source Threat Intelligence (TI) platforms like MISP, and countless
other ad-hoc and specialized sources (Gao, et al., 2021).
SA
For the sake of simplicity and ease of use for SOC teams, this program leverages
e
Th
framework and knowledge base encompasses the entire lifecycle of a cyber security
20
attack, from pre-cursory reconnaissance and initial exploit all the way through to
©
exfiltration and impact. The Enterprise ATT&CK framework also spans a comprehensive
array of platforms, including Windows, Linux, and Mac, plus cloud and service offerings
and containers, such that it applies to nearly any SOC organization (MITRE ATT&CK,
2021). In the ATT&CK matrix, delineated under each major phase of an attack are the
variety of techniques that have been observed being used by bad actors in real-world
security incidents. Each recognized attacker technique is also clearly articulated with
examples of real-world use by known adversary organizations and potential mitigation
actions.
© 2022
Internal UseThe SANS Institute
- Confidential Author retains full rights.
ts
Threat Hunting: This is the Way 12
gh
Each of those techniques is then uniquely identified and profiled with details
Ri
about the attack vector (MITRE ATT&CK, 2021). As explained in the recent journal
ll
Fu
article “An Efficient Approach of Threat Hunting Using Memory Forensics,” "Good
hunts rely on the hunter's talent to distinguish what data and tools are necessary to test the
ns
hypotheses” (Javeed, et al., 2020). The ATT&CK matrix provides this critical function of
ai
identifying the data sets and tools required for each hunting use case along with the TTPs
et
rR
observed, minimizing the need for manual intelligence research.
ho
The structure of each attack profile provides the SOC with a new organizational
ut
standard for identifying, discussing, and reporting on attacks, easing upward
,A
communication with business management. The matrix structure also provides an
inherent enterprise-wide measuring lens, allowing the SOC team to create various color-
te
itu
coded infographics. As shown in Figure 1 (MITRE, 2020), the matrix can be used to
demonstrate the perceived risk of various attack types to the organization, but it can also
st
In
represent priority levels, use case coverage, visibility levels, and other critical insights.
NS
This organizational-wide view of the SOC helps ensure all stakeholders clearly
understand the current security posture using empirical data, enabling them to discuss
SA
goals in a meaningful way. These reports enable much greater clarity and accuracy of
e
reporting for a SOC organization, which ultimately allows for better dialogue,
Th
© 2022
Internal UseThe SANS Institute
- Confidential Author retains full rights.
ts
Threat Hunting: This is the Way 13
gh
Ri
ll
Fu
ns
ai
et
rR
ho
ut
,A
te
itu
st
In
NS
Figure 1
SA
The threat hunting process relies heavily on the scientific method, developing a
hypothesis about a given attack, mapping out a logical structure of how it might have
22
occurred, then sifting through large amounts of data to validate and verify the hypothesis
20
(Wafula & Wang, 2019). For practical purposes, the approach to hunting begins very
©
simply: If we "assume the worst" – that an attacker has successfully conducted a given
attack – what would it look like?
© 2022
Internal UseThe SANS Institute
- Confidential Author retains full rights.
ts
Threat Hunting: This is the Way 14
gh
skills in SOC analysts and foster cross-organizational cooperation. The recommended
Ri
tasks will ultimately increase SOC effectiveness by improving the analysts' knowledge of
ll
Fu
network and endpoint data, understanding of data flows, situational awareness of the
enterprise, and broader business context. It will also improve inter-departmental
ns
communication and coordination with other IT organizations in the course of navigating
ai
various security findings.
et
rR
2.2.5. Output handling
ho
The output of threat hunting activities can be characterized as expected and
ut
unexpected. Expected output from threat hunting activities is defined here as those
,A
findings directly related to the hunting hypothesis. In contrast, unexpected output is
te
defined as other issues or events not directly associated with the original hypothesis.
itu
Incidents can (and will) be uncovered in the threat hunting process, and this represents
st
one of the most common forms of output. The number of incidents stemming from threat
In
hunting should become a measured and reported metric as a part of each organization’s
NS
threat hunting report package. This will help demonstrate the value of the threat hunting
program to the organization, supporting internal marketing efforts.
SA
Unexpected output frequently yields findings that are just as valuable as those
e
Th
related to the hypothesis. Examples of these cited by threat hunter James Pope include
violations of business rules such as cleartext data, misconfigurations of servers, and rogue
22
devices (Pope, 2021). Many of these kinds of issues provide an opportunity to shed light
20
organizational communication and highlighting process gaps. These findings should also
be tracked and reported as a part of the program's metrics.
© 2022
Internal UseThe SANS Institute
- Confidential Author retains full rights.
ts
Threat Hunting: This is the Way 15
gh
2.3. Technology
Ri
While this program identifies hunting actions to be leveraged against standard
ll
Fu
data sets, the practical ability to perform the recommended threat hunting tasks will vary
by organization, depending upon the actual event sources in use and the visibility tools
ns
deployed there. But SOC organizations who do not yet have a full complement of
ai
et
visibility tools should fear not – one of the key benefits of this program is the ability to
rR
identify visibility gaps and qualify the impact on the organization.
ho
Hunters rely on pervasive visibility into the enterprise's network, systems, and
ut
data and leverage a wide range of tools and systems to access and analyze the data. These
,A
systems include firewall logs, intrusion detection and prevention system logs, application
te
logs, operating system logs, and network traffic captures (Javeed et al., 2020).
itu
(Kokulu et al., 2019). In a recent survey published by ACM, "71.43% of analysts and
NS
60% of the managers" agreed that lack of visibility into critical data was the biggest
challenge their SOCs faced (Kokulu, et al., 2019). The issue is not only related to tool
SA
availability, however. Lack of visibility can also be introduced because of other teams
e
who may not follow organizational rules for inventory or configuration management,
Th
resulting in approved devices or configurations on the network. Again, threat hunting can
22
help uncover those rogue devices and nonstandard configurations and help close the
20
© 2022
Internal UseThe SANS Institute
- Confidential Author retains full rights.
ts
Threat Hunting: This is the Way 16
gh
security log data that is critical for organizational threat hunting efforts include Domain
Ri
Name Servers (DNS) servers, web proxy servers, Intrusion Detection Systems (IDS), and
ll
Fu
firewalls (Javeed, et al., 2020).
ns
For many, the SIEM system will represent the primary repository of
ai
organizational data for hunting, however logs alone are woefully insufficient for threat
et
hunting.
rR
2.3.2. Full packet capture
ho
As reported in recent years of SANS Threat Hunting surveys (Fuchs & Lemon,
ut
2020), the most common tool threat hunters need – and either did not have access to or
,A
could not access without a lot of difficulty – was network packet capture. Indeed, while
te
logs can be tampered with or erased (MITRE ATT&CK, 2021), full packet capture
itu
solutions can reveal additional details about attacker techniques and behaviors as they
st
were captured in transmission across the network. These solutions can help supplement
In
the enterprise's visibility gaps and support more advanced investigation and forensics
NS
2016).
e
visibility into end-user activities in the environment; typically agent-based, these tools
20
collect user-level data as well as kernel and operating system data for analysis and
comparison against baselines (Hurless, 2020). By capturing various categories of events
©
from the system, the management console can then be leveraged for additional
investigation and forensics activities across the broader enterprise and containment
actions against any infected hosts.
© 2022
Internal UseThe SANS Institute
- Confidential Author retains full rights.
ts
Threat Hunting: This is the Way 17
gh
3. Tier One Threat Hunting
Ri
Tier one hunting tasks focus on generating situational awareness about the
ll
Fu
operating environment, developing context, and validating business rules. These tasks
ns
were selected to build critical thinking skills and to cultivate curiosity and creativity–
consistent traits of threat hunters (Collins, 2018).
ai
et
3.1. Discovery
rR
Tier one hunting tasks will create a foundation for all other hunting activities
ho
performed in the SOC. These essential tasks will encompass a lot of data gathering and
ut
benchmarking to help to establish appropriate business context for the security
,A
environment: understanding the nature, structure, and goals of the business, as well as
te
where sensitive data resides and how it flows. Validating existing organizational policies
itu
is another critical baseline activity, such as the expected use of encryption and allowed or
st
disallowed protocols. Confirming or denying business rules are actually being enforced
In
helps eliminate assumptions (and ultimately risk), develop context about the production
NS
The list below serves as a recommended starting point for tier-one staff to use,
contemplate, and expand. Tier one analysts are encouraged to formally consider the scope
e
Th
of the enterprise they are charged with protecting, the reach of the network, and expected
traffic loads and types:
22
20
© 2022
Internal UseThe SANS Institute
- Confidential Author retains full rights.
ts
Threat Hunting: This is the Way 18
gh
•
Ri
Examine the traffic on the network and record the percentage of encrypted and
cleartext traffic.
ll
• Map out the protocols and services observed on the network, including ports used
Fu
and direction of the traffic.
• Identify all approved third-party connections to the network
ns
• Based on network observation, chart typical
hourly/daily/weekly/monthly/seasonal traffic patterns
ai
et
rR
ho
3.2. Baseline Development
ut
The output of the discovery activities outline in the previous section should then
,A
be used to develop a baseline for the SOC. By exploring and documenting the beginning
te
state of the network, asset inventory, and associated configurations (Collins, 2018), the
itu
SOC can begin to understand what is considered normal in the environment. With that
st
insight into expected patterns of activity, analysts can ultimately begin to recognize signs
In
of abnormal behavior. It is also critical to consider the breadth of the organization's entire
NS
attack surface and identify partial or poor visibility areas. Output should be documented,
version-controlled, and shared within the team.
SA
e
Tier two analysts will tackle more complex hunting tasks, building on the
22
activities conducted by tier one. In many cases, this will look a lot like system auditing.
20
As explained in the EEE journal article titled “Enabling Efficient Cyber Threat Hunting
©
The sections that follow serve as a recommended starting point for tier two staff
to consider and expand. Tier two analysts are encouraged to closely examine critical
systems, focusing on those identified by the discovery activities as serving essential
production functions, housing sensitive data, or with heightened exposure like facing the
© 2022
Internal UseThe SANS Institute
- Confidential Author retains full rights.
ts
Threat Hunting: This is the Way 19
gh
internet. This is sometimes referred to as the "crown jewel" approach (Lee & Lee, 2016),
Ri
wherein hunters consider the organization's most important data, where it resides, and the
ll
Fu
most likely scenarios an attacker would use to get them. Tier two analysts may also
receive issues or questions uncovered by tier one analysts in their threat hunting tasks,
ns
enabling collaboration for interesting findings.
ai
et
4.1.1. Review of Sensitive Servers
rR
Tier two analysts should verify services running, open ports, routing tables, and
ho
DNS entries for all critical servers, and review local, service, and administrator accounts
ut
assigned to each. This higher skill level resource should also review the environment for
,A
unauthorized use of Powershell. Powershell is a powerful scripting and interactive
te
command-line-interface tool commonly found in large Windows environments (MITRE,
itu
2020). Given its capabilities, it is imperative to ensure that Powershell is only in use on
st
for the SOC. Tier two resources should review the network to understand how traffic
flows from within the internal network out to the internet. Documenting network address
e
Th
blocks in use, proxy configurations, and network segmentation is not glamourous work
but provides essential context for threat hunting. It is important to understand what
22
applications and protocols are observed coming inbound into the enterprise’s critical
20
servers, and looking for port and service mismatches can also yield interesting results.
©
Given its ubiquitous nature and potential for exploitation, review HTTP and DNS
traffic in detail. Look for unusual artifacts inconsistent with modern web usage, such as
user-agent strings associated with old or non-standard browsers. Other artifacts of interest
include those that are not indicative of human activity, such as an HTTP direct to IP
requests (RSA, 2021).
© 2022
Internal UseThe SANS Institute
- Confidential Author retains full rights.
ts
Threat Hunting: This is the Way 20
gh
or in parts. In terms of use cases, insider threat is a tough challenge for detection
Ri
(Detecting and Identifying Insider Threats, 2021), making it an ideal candidate for
ll
Fu
proactive hunting methodologies. Automated tools are beginning to show promise for the
detection of insider threats using machine learning and, more recently, "deep learning,"
ns
which layers on top of machine learning another model derived from complex data (Yuan
ai
& Wu, 2021). However, because insider threat activities inherently contain legitimate
et
rR
work actions, it can be difficult to separate authorized activities from malicious ones.
Threat hunting is aided in this use case by user behavior analysis tools.
ho
ut
Process hollowing is one specific type of process injection technique within the
,A
ATT&CK framework (MITRE ATT&CK, 2020); this can be thought of in general terms
as a bad actor leveraging a legitimate process on a machine to inject malicious code or
te
itu
processes. Process hollowing attacks are particularly difficult for automatic detection use
cases, so that makes this technique an exciting use case for hunting (MITRE ATT&CK,
st
In
2020). Finally, a lot of tier three hunting activities seem to follow a notion that something
NS
"just doesn't seem right” (Wyler, 2021). Some experts have warned that those early threat
hunting days will result in chasing a lot of false leads, but the examination process has
SA
great merit of its own. As such, analysts should be encouraged to follow their instincts!
e
Th
are tracked and reported. In most SOC organizations, this means opening a ticket in the
20
organization's ticketing system, which will identify the asset(s) involved, as well as other
©
technical artifacts such as IP address and hostname, then track the investigation and
resolution of the issue. Other findings may warrant remediation on a wide variety of
systems. Those configuration changes should also be tracked in accordance with the
SOC's established ticket management procedures. Finally, some outcomes warrant
creating or updating procedural documentation, which should be tracked to resolution.
5. Analysis
This section provides ideas for metrics, reports, and infographics that the SOC
organization can use to help track and measure the Threat Hunting program's gains.
© 2022
Internal UseThe SANS Institute
- Confidential Author retains full rights.
ts
Threat Hunting: This is the Way 21
gh
5.1. Metrics
Ri
• Number of security controls validated
ll
Fu
• Percentage of traffic unencrypted vs encrypted
ns
• Number of MITRE Tactics, Techniques, and Procedures (TTP) IDs validated
ai
et
• Number of users/applications sending and receiving data in cleartext
rR
• Number of misconfigurations or errors found
ho
• Overall time invested
ut
,A
5.2. Reports
•
te
Use case confidence report - identify use cases on ATT&CK matrix in RED that
itu
cannot be validated due to lack of visibility. Use GREEN for those that have been
st
• Visibility gap report – identify use cases on ATT&CK matrix in RED that cannot
NS
be validated due to lack of visibility. Use GREEN for those that have been
SA
validated and YELLOW for those with limited visibility or difficulty in obtaining.
• Trends – track hunting metrics over time and develop longer-term checkpoints to
e
Th
ensure that overall time invested is still yielding results. If not, use case
adjustments must be made, and the tier three team should initiate that review and
22
revise appropriately.
20
©
© 2022
Internal UseThe SANS Institute
- Confidential Author retains full rights.
ts
Threat Hunting: This is the Way 22
gh
6. Implications for Future Research
Ri
ll
Fu
6.1.1. Rationale for Project
ns
The genesis of this project was spawned by many years of professional experience
ai
and observation in the cyber security industry focused on helping security organizations
et
improve their defensive security posture. Despite decades of continuous development of
rR
new security tools and products in the cyber security community, attackers are still
ho
winning while threat detection and response capabilities are lag behind.
ut
The focused goal was of the project was twofold. The first goal was to break
,A
down the technical barriers to entry for tackling threat hunting as a discipline, which was
te
addressed by prescribing specific threat hunting activities to be performed by SOC
itu
analysts of every tier and enabling them to conduct them. The secondary goal was to
st
eliminate any remaining excuses or misconceptions about the value of threat hunting. As
In
demonstrated here, threat hunting is one of the best ways to bridge the lingering detection
NS
gap, and it does not require seasoned subject matter experts with a formal Threat Hunter
SA
title or years of training for existing personnel. The desired outcome for this project was
an introductory threat hunting program framework that would be easy for most any
e
Th
modern SOC organization to implement and use with no capital expenditures and only
existing staff.
22
20
I initially set out to provide very specific techniques to hunt for in the
environment, plus scripts or tools that could be used to test it, but as I got further into the
©
project, I found that approach much too restrictive. I instead focused more on the process
with specific examples that would apply to a broader range of organizations. This
represents another possible area for future research and follow-up for the community.
© 2022
Internal UseThe SANS Institute
- Confidential Author retains full rights.
ts
Threat Hunting: This is the Way 23
gh
upon existing ones. Of particular interest are the gains demonstrated in using the matrix
Ri
as a communication and reporting tool.
ll
Fu
6.1.3. Automation
ns
Automating of some underlying activities for threat hunting represents a level of
ai
maturity that some SOCs may wish to pursue over time. Automating tasks can be as basic
et
as using small scripts to kick off data collection or as complex as running several
rR
sequential or related activities before manual review. As characterized in a recent journal
ho
article titled “An Efficient Approach of Threat Hunting Using Memory Forensics,”
ut
because threat hunting relies "deeply on mechanization and machine support, the
,A
procedure itself can't be completely automatic nor can any invention accomplish hunting
te
for an expert(Javeed et al., 2020). Threat hunting will likely continue to require manual
itu
oversight with sophisticated skills, but modern Security Orchestration and Automation
st
(SOAR) platform provide an entry point for integrating threat intelligence with
In
Countless tools, programs, and otherwise good ideas fail simply due to poor
"marketing." As cyber security professionals, we are sometimes so technology-focused
e
Th
that we fail to remember any significant change – such as introducing a new tool or
process – needs a conscientious introduction, frequent refresher communication, and
22
marketing. Everyone in the SOC should be encouraged to embrace their new role as a
hunter and serve as a spokesperson for the program within the broader organization.
Launch the program by telling each major stakeholder what this program will do
for them. For business executives, it will reduce risk, provide more quantitative insight
into the SOC's operation, and have a demonstrated return on investment (ROI). For SOC
managers, it is a powerful training, development, and retention tool. For SOC analysts –
at least most of us – it's really, truly a lot of fun.
© 2022
Internal UseThe SANS Institute
- Confidential Author retains full rights.
ts
Threat Hunting: This is the Way 24
gh
7. Conclusion
Ri
ll
This project sought to create an easily consumable threat hunting program that
Fu
could be leveraged by SOC managers using only their existing personnel and tools. The
ns
program was subsequently structured as a flexible, easy-to-implement, customizable
ai
framework that can be implemented by most any modern SOC. The goal was to
et
demonstrate the value of a threat hunting program to the SOC's employee satisfaction and
rR
retention goals, as well the benefits to the organization's security posture, so that any
ho
SOC managers not yet threat hunting will realize that this is the way.
ut
,A
te
itu
st
In
NS
SA
e
Th
22
20
©
© 2022
Internal UseThe SANS Institute
- Confidential Author retains full rights.
ts
Threat Hunting: This is the Way 25
gh
References
Ri
ll
(ENISA), E. U. (2020, December). How to Setup CSIRT and SOC. Retrieved from ENISA:
Fu
https://www.enisa.europa.eu/publications/how-to-set-up-csirt-and-
ns
soc/at_download/fullReport
ai
et
Akinrolabu, O., Agrafiotis, I., & Erola, A. (2018). The Challenge of Detecting Sophisticated
rR
Attacks: Insights from SOC Analysts. Hamburg, Germany: ARES 2018: Proceedings of
ho
the 13th International Conference on Availability, Reliability, and Security.
ut
Baykara, S. (2021, January 28). What is the Separation of Duties Principle and How is it
,A
Implemented? Retrieved from PCI DSS Guide: https://www.pcidssguide.com/what-is-
te
itu
the-separation-of-duties-principle-how-is-it-implemented/
st
Bhardwaj, A., & Goundar, S. (2019). A framework for effective threat hunting. Network Security,
In
hunting/9781492028260/.
e
Crowley, C. (2019). Common and Best Practices for Security Operations Centers: Results of the
Th
program/common-practices-security-operations-centers-results-2019-soc-survey-
20
39060.pdf
©
Daszczyszak, R., Ellis, D., Luke, S., & Whitely, S. (2019). TTP-Based Hunting.
https://www.mitre.org/sites/default/files/publications/pr-19-3892-ttp-based-hunting.pdf:
MITRE.
Fuchs, M., & Lemon, J. (2020). SANS 2020 Threat Hunting Survey Results.
https://www.sans.org/white-papers/40020/: SANS.
Gao, P., Shao, F., Liu, X., Xiao, X., Qin, Z., Xu, F., . . . Song, D. (2021). Enabling Efficient
Cyber Threat Hunting With Cyber Threat Intelligence. IEEE 37th International
© 2022
Internal UseThe SANS Institute
- Confidential Author retains full rights.
ts
Threat Hunting: This is the Way 26
gh
Conference on Data Engineering (ICDE), 193-204,
Ri
ll
https://doi.org/10.1109/ICDE51399.2021.00024.
Fu
Hranický, R., Breitinger, F., Ryšavý, O., Sheppard, J., Schaedler, F., Morgenstern, H., & Malik,
ns
S. (2021). What do incident response practitioners need to know? A skillmap for the
ai
years ahead. Forensic Science International: Digital Investigation, Vol 7,
et
rR
https://doi.org/10.1016/j.fsidi.2021.301184.
ho
Hurless, C. (2020). Open-Source Endpoint Detection and Response with CIS Benchmarks,
ut
Osquery, Elastic Stack, and TheHive. https://www.sans.org/white-
,A
papers/39900/?utm_medium=Print&utm_source=SANS%20EDU%20Newsletter&utm_c
te
ampaign=Research%20Review%202020: SANS.
itu
Javeed, D., Khan, M. T., Ahmad, I., Iqbal, T., Badamasi, U. M., Ndubuisi, C. O., & Umar, A.
st
In
https://www.proquest.com/openview/f4a78001c4cad853c35f05c79df125af/1?pq-
e
origsite=gscholar&cbl=2044553.
Th
Kazmierczyk, J., Romashkina, G. F., & Macholak, P. (2020). Lifeflong Learning as an Employee
22
http://doi.org/10.9770/jesi.2020.8.1(71).
©
SANS.
Kokulu, F. B., Soneji, A., Bao, T., Shoshitaishvili, Y., Zhao, Z., Doup\'{e}, A., & Ahn, G.-J.
https://doi.org/10.1145/3319535.3354239.
© 2022
Internal UseThe SANS Institute
- Confidential Author retains full rights.
ts
Threat Hunting: This is the Way 27
gh
Lee, R. M., & Lee, R. (2016). The Who, What, Where, When, Why, and How of Effective Threat
Ri
ll
Hunting. https://www.sans.org/white-papers/who-what-where-when-why-how-effective-
Fu
threat-hunting/: SANS.
ns
Lee, R. M., & Lee, R. T. (2018). SANS 2018 Threat Hunting Survey.
ai
https://www.malwarebytes.com/resources/files/2018/09/survey_threathunting-
et
rR
2018_malwarebytes.pdf: SANS.
ho
Mallon, M. (2021, August 3). <placeholder>. (M. Raney, Interviewer)
ut
MITRE. (2020, June 24). Command and Scripting Interpreter: PowerShell. Retrieved from
,A
MITRE ATT&CK: https://attack.mitre.org/techniques/T1059/001/
te
MITRE ATT&CK. (2020, November 10). Process Injection: Process Hollowing. Retrieved from
itu
MITRE ATT&CK. (2021, April 21). Enterprise Matrix. Retrieved from MITRE ATT&CK:
NS
https://attack.mitre.org/matrices/enterprise/
SA
MITRE ATT&CK. (2021, April 24). MITRE ATT&CK. Retrieved from Indicator Removal on
e
Host: https://attack.mitre.org/techniques/T1070/
Th
i5-13.
©
Oltsik, J., & Alexander, C. (2020). The Life and Times of Cybersecurity Professionals 2020. RSA
prd.lanyonevents.com/published/rsaus20/sessionsFiles/18313/2020_USA20_PART1-
R07_01_4th-Annual-Life-and-Times-of-a-Cybersecurity-Pro-Is-It-Getting-Better.pdf.
Podzins, O., & Romanovs, A. (2019, April). Why SIEM is Irreplaceable in a Secure IT
Pope, J. (2021, July 28). Principal Threat Hunter. (M. Raney, Interviewer)
© 2022
Internal UseThe SANS Institute
- Confidential Author retains full rights.
ts
Threat Hunting: This is the Way 28
gh
RSA. (2021). RSA NetWitness Hunting Guide. Retrieved from RSA:
Ri
ll
https://community.rsa.com/t5/netwitness-platform-threat/rsa-netwitness-hunting-guide/ta-
Fu
p/564743
ns
Steil, A. V., de Cuffa, D., Iwaya, G. H., & Pacheco, R. C. (2020). Perceived Learning
ai
Opportunities, Behavioral Intentions and Employee Retention in Technology
et
rR
Organizations. Journal of Workplace Learning, ISSN: 1366-5626.
ho
Tatam, M., Shanmugam, B., Azam, S., & Kannoorpatti, K. (2021). A review of threat modelling
ut
approaches for APT-style attacks. Heliyon, Volume 7, Issue 1,
,A
https://doi.org/10.1016/j.heliyon.2021.e05969.
te
Wafula, K., & Wang, Y. (2019). CARVE: A Scientific Method-Based Threat Hunting Hypothesis
itu
Wyler, N. (2021, July 22). Principal Threat Hunter. (M. Raney, Interviewer)
SA
Yuan, S., & Wu, X. (2021). Deep learning for insider threat detection: Review, challenges and
e
© 2022
Internal UseThe SANS Institute
- Confidential Author retains full rights.