U LT I M AT E G U I D E T O

Penetration
Testing
TA B L E O F C O N T E N T S

3 Addressing The Cybersecurity Crisis

4 Why Penetration Testing?
7 Penetration Testing Options: Pros and Cons
10 Traditional Penetration Tests and Low Expectations
13 Rise of Penetration Testing as as Service
14 What’s Next for Penetration Testing?
15 The Bugcrowd Security Knowledge Platform
18 Dawn of a New Era
ADDRESSING THE CYBERSECURITY CRISIS
Cybersecurity and digital transformation have
always gone hand-in-hand. But with the
blistering rate of transformation fueled by the
COVID-19 pandemic, cybersecurity now ranks
behind only climate change as a global risk in
the minds of risk managers and consumers
alike (AXA Future Risks Report 2021).
Even so, in most organizations, fragmented
and short-staffed cybersecurity environments
leave critical systems vulnerable to cyber
attacks, putting customers, partners, and
reputations at risk. Although executives are
constantly reminded of this risk posed by
events such as the SolarWinds hack, the
Colonial Pipeline ransomware incident, and the
Log4j vulnerability, 81% of them believe that the
costs and constant effort required to stay ahead
of attackers is “unsustainable” (A e ture).
In this environment, penetration testing remains
a critical tool in the security leader s toolbox.
The implementation details matter, however
For too long, the industry has relied on a
cumbersome, consulting-heavy approach that
does little to reduce risk. For that reason,
traditional approaches to pen testing have
become part of the problem, not part of the
solution.

IN THIS GUIDE YOU WILL LEARN...

• Why penetration testing is done today
• Current approaches to penetration testing,
with pros and cons
• Why the traditional approach comes up short
• The Rise of Penetration Testing as a Service
• What crowdsourcing brings to penetration
testing
• How the Bugcrowd Platform enables
crowdsourced Penetration Testing as a
Service and other security testing strategies

U LT I M A T E G U I D E T O P E N E T R A T I O N T E S T I N G | 3
 W H Y P E N E T R AT I O N T E S T I N G ?
Penetration testing has been with us for a long
time in one form or the other, but adoption has
been accelerating as of late, with artner
estimating a total market size of 4. B by
202 . (And that s just for commercial tools use
of open source tools is also significant.)
p until recently, compliance (e.g., for PCI-
DSS) was the dominant driver for pen testing.
Today, per industry research, 9% of adopters
do pen tests to assess security posture, and
% do them for compliance purposes--a
much more even split, and a signal that many
orgs do them for both reasons.
Compliance can be an opportunity to secure
investment for penetration testing in
organizations with less mature cybersecurity
practices. But annual or biannual compliance-
driven testing alone is just table stakes in most
companies there are many other important
reasons to invest in penetration testing.

Per the ational Institute of Standards and Technology ( IST)

Penetration testing is security testing in which assessors

mimic real-world attacks to identify methods for
circumventing the security features of an application,
system, or network.

For example, the continuous development cycles typical of cloud-based environments cry out
for more frequent, if not continuous, testing. And the turmoil created by mergers and
acquisitions, particularly in regulated industries, is a common driver for more rigorous testing
than what checking a compliance checkbox will provide. (See next section for more
examples.)

With the increasing complexity of the attack surface, which has expanded well beyond web
apps, networks, and databases to include APIs, cloud infra, and even physical devices, the
drivers for doing deep penetration testing are certain to multiply.

4 | U LT I M A T E G U I D E T O P E N E T R A T I O N T E S T I N G
REDUCE CYBER RISK
As cybersecurity becomes more of a board-
level issue, it is increasingly being defined in
the language that boards are most familiar with,
that of business risk. This risk is being more
accurately defined in terms of probability and
cost of remediation. For organizations with a
mature cybersecurity function, managing cyber
risk is the top priority in cyber defense, and
a robust testing regime allows companies to
directly reduce this risk.
Learn more about this in
The Ultimate Guide to Cyber Risk Management.
SATISFY STAKEHOLDER REQUIREMENTS
Stakeholders such as customers, suppliers,
investors, and regulators have a considerable
role in an organization’s decision-making.
The most obvious place where this plays is
in supply chain risk, where key stakeholders
need to be reassured that a supply chain is
sustainable, secure, and free of criminality.
During the pandemic, supply chains came under
considerable pressure, and penetration testing
played a pivotal role in helping organizations
U LT I M A T E G U I D E T O P E N E T R A T I O N T E S T I N G | 5
Penetration testing identifies vulnerabilities
and threats that represent tangible business
risk. This could be in the form of regulatory risk
from improperly protected customer data, risk
associated with the rapidly growing ransomware
complex, or even risk to intellectual property from
Advanced Persistent Threats at the nation-state
level.
adapt to these challenges and protect customer
and partner data.
Stakeholders have also adapted to these
changing needs of penetration tests, such as
in the UK, where the National Cybersecurity
Centre added a home and remote-working
exercise to its existing package of penetration
testing exercises.
 PRESERVE THE ORGANIZATION’S IMAGE AND REPUTATION

Cyber incidents cause fundamental harm to an organization’s reputation, particularly when

they put customer data at risk and result in prolonged legal proceedings. Breaches and attacks
are becoming more prevalent in business reporting, and consumers are more wary about their
data and privacy. Penetration tests represent a crucial part of the cybersecurity stack and help
prevent these attacks and the resultant harm to reputation.

6 | U LT I M A T E G U I D E T O P E N E T R A T I O N T E S T I N G
 P E N E T R AT I O N T E S T I N G O P T I O N S
PROS AND CONS

Although the tools and tactics used by pentesters themselves don t vary much, the testing
frameworks within which pentesters operate have significant differences. The framework you
choose will have a major impact on testing experience for everyone involved (testers as well
as testing consumers).

TRADITIONAL ("STATUS QUO") PENETRATION TESTING

We ll go into more detail in the next section about how the most common approach to pen
testing has led to low expectations for pen testing, but at a high level, pros and cons include

PROS
• Established budget line item
• A known quantity
• Best suited for targets that require
physical presence to access/test
CONS
• Delays to scheduling and results
• Inflexible with questionable skill fit
• Not optimized to incentivize
true risk reduction

CROWDSOURCED PENETRATION TESTING

This model implies the use of a bench of trusted, pay-per-project testers who are crowdsourced
from the massive security researcher community. Crowdsourced testing is quickly becoming the
top choice for organizations seeking more impact from penetration testing.

PROS CONS
• Offers access to the • Not optimized for highly sensitive
massively diverse skill sets of or physical targets too big to ship
a global community • Still unfamiliar to many AppSec
• Option to pay for impact decision makers
instead of time to incentivize • New business case may be
better results required
• Enables easy tester rotation

| U LT I M A T E G U I D E T O P E N E T R A T I O N T E S T I N G
INTERNAL SECURITY TESTING

While often infeasible for smaller organizations, some enterprises prefer to build and maintain
in-house teams ( red teams ) of security testing. This approach allows the organization to set
its own schedule and may reduce barriers in some areas, (e.g., provision of credentials).

PROS CONS
• Best for extremely sensitive work • Labor-intensive to set up and
(e.g., Secret, NOFORN) maintain
• Tests can be run as • Impossible to retain all possible
frequently as needed testing skills
• Little marginal cost to testing • Hard to acquire new skills
when needed

U LT I M A T E G U I D E T O P E N E T R A T I O N T E S T I N G | 8
 A MIXED-TESTING APPROACH

Some organizations use a combination of traditional, crowdsourced, and internal

testing to meet the specific needs of each project.

PROS CONS
• Includes the best aspects • Includes the worst aspects
of each method of each method
• Potential for thorough security • Complex to arrange and
coverage maintain
• Testing depth is as-needed • (Potentially) extremely
for each project high-cost

9 | 2 0 2 2 U LT I M A T E G U I D E T O P E N E T R A T I O N T E S T I N G
 T R A D I T I O N A L P E N E T R AT I O N T E S T S
AND LOW EXPECTATIONS
When we first released our guide to
penetration testing in 2020, there was already
a growing consensus that the most traditional
approaches to testing were dated, if not
obsolete. These traditional penetration tests
adopt a “one-size-fits-all” approach:
simulated attacks carried out by one to two
testers, who offer box-ticking results
according to narrowly defined, compliance-
based methodologies.
GAPS IN THE TRADTIONAL MODEL
SLOW LAUNCHES
Tests can take months to schedule, due to
resource constraints on the part of testing
providers and their desire to reduce time on
the “bench” for salaried employees.
This might seem fine to companies that
consider these tests as the equivalent of a
routine dental check-up but not the many
organizations that worry that they may need
an emergency root canal.
10 | U LT I M A T E G U I D E T O P E N E T R A T I O N T E S T I N G
They can be useful to confirm hypotheses
or concerns within the organization, but they
do not meaningfully reduce risks or address
known unknowns.
Since then, gaps and failings in the strict and
narrow approach to penetration testing
have resulted in even lower expectations for
penetration testing from its adopters. Here
are the most pressing concerns.
Many of these tests also have strictly limited
time windows to deliver a testing schedule.
These can rule out some crucial testing
methods—for example, it is impossible
to conduct a 10-day scan as part of an
assignment with an allocation of five days
of testing. Putting artificial time constraints
on penetration testing reduces the extent to
which it can reduce risk.
 DELA ED ESULTS
Another way that timing is a problem is the
delay in receiving results. With a standard
penetration test, the customer doesn’t receive
results until the engagement is concluded,
often 14–24 days after testing begins. This
leaves tested assets vulnerable for an
unnecessarily long time, which can be a real
issue when the penetration test is needed to
address a newly identified risk as quickly as
possible.
PROBLEMS WITH SKILLS FIT AND APPLICATION
A traditional penetration test is carried out by
one to two testers over a period of two weeks.
Regardless of how experienced the testers
are, they can’t be versed in every possible
attack technique, and their skill sets may not
be appropriate for the asset being tested.
Equally, customers don’t have the option to
select which testers are assigned to their
projects. Paying for these tests “off the shelf”
adds a randomized element around what
testers the organization has access to, which
can have a profound effect on the results.
1 1 | U LT I M A T E G U I D E T O P E N E T R A T I O N T E S T I N G
Most digital assets are only penetration tested
a maximum of one to two times per year. With
modern, agile development lifecycles, new
codebase versions are released much more
frequently. While an asset may be secure
immediately following a test, new code
releases could leave it vulnerable to attack
until the next scheduled test.
There is also an issue of the skills being
applied too narrowly, with most penetration
tests being checklist-based. These provide
minimal time or incentive for testers to use
their initiative or “dig deeper” to find complex
vulnerabilities. This is exacerbated by a “pay
for time” business model, where buyers pay
for a certain number of tester hours and
the testers are only required to finish the
methodology in that time. The number and
severity of vulnerabilities that surface during
this time is irrelevant to the tester’s final pay.
 LOW-IMPACT FINDINGS

All the abovementioned limitations contribute to the central problem with relying on traditional
penetration tests in isolation. The narrow nature of the timing, skill sets, compliance focus, and
selection of participants reduces the effectiveness of an engagement relative to alternatives.

Due to poor results, high cost, and time delays, traditional

penetration testing services are not a c ost-effective security
control. Worse, because skill fit is likely suboptimal and
testers aren’t incentivized to “go deep,” it’s likely that
high-risk vulnerabilities will be missed.

Given this, the traditional penetration testing model is simply not suited
to the needs and goals of most adopters today.

1 2 | U LT I M A T E G U I D E T O P E N E T R A T I O N T E S T I N G
 R I S E O F P E N E T R AT I O N T E S T I N G A S
A SERVICE
With the new dominance of cloud in IT,
recently we've seen the emergence of
Penetration Testing as a Service (PTaaS)
options that modernize pen testing by
bringing "XaaS" agility, scale, and user
experience. This is a welcome development
for buyers accustomed to the cumbersome,
consulting-heavy approaches of traditional
vendors.
KEY BENEFITS OF PTAAS
• Brings modern SaaS sensibilities to pen
testing -- such as self-service dashboards,
repeatability/scale, and a good user
experience for pentesters and adopters
alike
• Enables much faster launches (days
instead of weeks) and report delivery than
traditional approaches
• Findings are integrated directly with
DevSec workflows so remediation can
begin fast
1 3 | U LT I M A T E G U I D E T O P E N E T R A T I O N T E S T I N G
That said, because most PTaaS options rely
heavily on automation to achieve scale, their
testing lacks the depth and intensity that only
human-driven testers can provide. As a result,
adopters should careful to validate that their
PTaaS vendor offers more than a vulnerability
scan with a pretty dashboard on top.
BUT WATCH OUT FOR....
• Excessive reliance on automation
that leads to shallow/checkbox
results
• Limited choice of target types
• Manual scoping
• Narrow, siloed solutions that don't
integrate with other programs
• "Crowd washing" - old-fashioned
pentester sourcing masquerading as
crowdsourcing
 W H AT ’ S N E X T F O R
PENETRATION TESTING?
What should we learn from all this? That the most
ONLY BUGCROWD PTAAS
effective, convenient way to do pen testing is to bring
OFFERS:
the value of crowdsourcing to Penetration Testing as a
Service--what Bugcrowd calls "penetration testing
• A trusted and expert team of
done right."
pentesters selected for your specific
needs
CROWD-POWERED PTAAS
• 24/7 visibility into timelines, analytics,
While many organizations share a need for compliance, prioritized findings, and pentester
not all have the same testing requirements or capacity. progress through the methodology
Some seek continuous coverage to match increasingly • Ability to "clone" pen tests at scale for
rapid development cycles. Others need shorter testing repeatability and manage them all as a
windows throughout the year, as dictated by group
engineering workflows or budgetary and procurement
cycles. Equally, an organization 's appetite for tester • Easy rotation of the pentester bench as
incentivizes may be shaped by its bandwidth to needed
address vulnerabilities and its ability to maintain an • A choice of "pay-for-time" or "pay-for-
elastic pool of monetary rewards. impact" incentives
• Combined, these factors enable crowd-
To address these varied needs, Bugcrowd provides
powered penetration tests to identify
crowd-powered PTaaS on its Security Knowledge
on average 7X more high-priority
PlatformTM--matching skill sets from the global security
vulnerabilities than traditional
researcher community (the Crowd) for high-impact
penetration tests
results, while providing methodology-based coverage
and compliance reporting.

14 | U LT I M A T E G U I D E T O P E N E T R A T I O N T E S T I N G
 THE BUGCROWD SECURITY KNOWLEDGE
PLATFORM
PTaaS isn't the only way to leverage the power of the Crowd. The multi-solution Bugcrowd Security
Knowledge PlatformTM brings the right crowd into all your workflows at the right time, allowing you to
run bug bounties, penetration tests, vulnerability disclosure programs, and more at scale and in an
integrated, coordinated way.

Best Security ROI Instant Focus on Contextual Intelligence Continuous, Resilient

from The Crowd Critical Issues for Best Results Security for DevOps

We match you with the right trusted Working as an extension of the platform, We apply accumulated knowledge The platform integrates workflows with
security researchers for your needs our global security engineer team rapidly from over a decade of experience your existing tools and processes to
and environment across hundreds validates and triages submissions, with across 1000s of customer solutions ensure that applications and APIs are
of dimensions using ML P1s often handled within hours to your goals for better outcomes continuously tested before they ship

1 5 | U LT I M AT E G U I D E T O P E N E T R A T I O N T E S T I N G
MANAGED VDPs
Bugcrowd VDPs provide a coordinated channel
and framework to enable anyone, anywhere, to
responsibly disclose security vulnerabilities
found in publicly accessible assets. Bugcrowd’s
fully managed approach reduces noise, and
accelerates remediation.
• Demonstrate Security Maturity: Build
stakeholder confidence and trust by
protecting digital assets and responding to
known risks
• Formalize Security Feedback: Create
a channel for security feedback and a
framework to manage vulnerabilities
discovered by researchers
• Meet Compliance Requirements: Align
cybersecurity programs with best practices,
as defined by the US Government, NIST,
DOJ, FDA, and others
MANAGED BUG BOUNTY
Managed Bug Bounty solutions on the Bugcrowd
Platform level the playing field by combining
data, technology, and the ingenuity of the Crowd
at scale to continuously find more critical
vulnerabilities in your attack surface than other
approaches.
• Instant Impact on Risk and Security Posture:
A team of skilled, trusted researchers
precisely matched to your needs
continuously find hidden vulnerabilities that
other approaches miss
• Proven Security ROI: A results-driven model
ensures you pay only for vulnerabilities that
present a risk, and not the time or effort it
took to find them
• Low Overhead: Our platform-powered,
managed solution seamlessly integrates with
your development and security processes,
delivering frictionless setup
ATTACK SURFACE MANAGEMENT
Bugcrowd’s Attack Surface Management solution
helps organizations reduce risk from unknown, or
unprioritized assets that often become a primary
target for attackers. Asset recon experts hunt for
unseen assets, while a software-based solution
continually scans for new connections and activity.
• Reduce Unknown Attack Surface: Find
forgotten assets that scanners can’t. Receive
priority risk-ranking based on each asset’s
potential for vulnerability
• Continuous Scanning: Asset Inventory
leverages a pre-indexed snapshot of the
internet which continues to grow. New
assets are added to your inventory as they
are discovered and attributed
• Live Alerting: Receive alerts on high-risk
events like open ports, or soon to be expired
security certificates. Share findings with
external teams like Marketing, Sales, and
Product to ensure quick fixes

17 | U LT I M A T E G U I D E T O P E N E T R A T I O N T E S T I N G
DA N O A N RA

Some security leaders be get nostalgic about
the traditional approach to penetration testing--
it's comfortable and familiar. But adoption of
Bugcrowd's crowdsourced PTaaS shows that
the trend is toward adoption of more modern,
distributed testing that creates access to
diverse skill sets, and away from cumbersome,
consulting-heavy approaches that depend on
scanning or plain-vanilla human testing.
Even for organizations that prioritize
compliance over risk reduction in penetration
testing, crowdsourcing can be as good, or
better, at meeting compliance requirements
than a small team.
Ultimately, penetration testing is another piece
of the security puzzle. Organizations should use
it with a combination of security tooling and
processes to find and remediate vulnerabilities
in the software development lifecycle (SDLC).
Crowdsourced penetration testers are a crucial
piece of this dynamic security puzzle. As they
continue to build out this industry, expect it to
continue to grow in importance and adoption.

Curious to learn how your organization can use

Bugcrowd Penetration Testing? Learn more:
bugcrowd.com/products/oen-test-as-a-service
2 0 2 2 U LT I M A p E G U I D E T O P E N E T R A T I O N T E S T I N G | 1 9