Empirical Research for Software

Security Foundations and Experience

1st Edition Ben Othmane
Visit to download the full and correct content document:
https://textbookfull.com/product/empirical-research-for-software-security-foundations-
and-experience-1st-edition-ben-othmane/
More products digital (pdf, epub, mobi) instant
download maybe you interests ...

Empirical Philosophical Investigations in Education and

Embodied Experience Joacim Andersson

https://textbookfull.com/product/empirical-philosophical-
investigations-in-education-and-embodied-experience-joacim-
andersson/

Software Development Pearls Lessons from Fifty Years of

Software Experience 1st Edition Wiegers

https://textbookfull.com/product/software-development-pearls-
lessons-from-fifty-years-of-software-experience-1st-edition-
wiegers/

Contemporary Empirical Methods in Software Engineering

Michael Felderer

https://textbookfull.com/product/contemporary-empirical-methods-
in-software-engineering-michael-felderer/

Foundations of Modern Macroeconomics 3rd Edition Ben J.

Heijdra

https://textbookfull.com/product/foundations-of-modern-
macroeconomics-3rd-edition-ben-j-heijdra/
Empirical Research in Statistics Education 1st Edition
Andreas Eichler

https://textbookfull.com/product/empirical-research-in-
statistics-education-1st-edition-andreas-eichler/

Fuzzing for Software Security Testing and Quality

Assurance Ari Takanen

https://textbookfull.com/product/fuzzing-for-software-security-
testing-and-quality-assurance-ari-takanen/

Foundations of Software Engineering Ashfaque Ahmed

https://textbookfull.com/product/foundations-of-software-
engineering-ashfaque-ahmed/

Introduction to Medical Software Foundations for

Digital Health Devices and Diagnostics 2nd Edition
Papademetris

https://textbookfull.com/product/introduction-to-medical-
software-foundations-for-digital-health-devices-and-
diagnostics-2nd-edition-papademetris/

Responsible Lobbying Conceptual Foundations and

Empirical Findings in the EU 1st Edition Theresa Bauer
(Auth.)

https://textbookfull.com/product/responsible-lobbying-conceptual-
foundations-and-empirical-findings-in-the-eu-1st-edition-theresa-
bauer-auth/
 Empirical Research
for Software Security
Foundations and Experience
 CRC Series in Security, Privacy and Trust
SERIES EDITORS
Jianying Zhou Pierangela Samarati
Institute for Infocomm Research, Singapore Università degli Studi di Milano, Italy
jyzhou@i2r.a-star.edu.sg pierangela.samarati@unimi.it

AIMS AND SCOPE

This book series presents the advancements in research and technology development
in the area of security, privacy, and trust in a systematic and comprehensive manner.
The series will provide a reference for defining, reasoning and addressing the security
and privacy risks and vulnerabilities in all the IT systems and applications, it will mainly
include (but not limited to) aspects below:
• Applied Cryptography, Key Management/Recovery, Data and Application Security and Privacy;
• Biometrics, Authentication, Authorization, and Identity Management;
• Cloud Security, Distributed Systems Security, Smart Grid Security, CPS and IoT Security;
• Data Security, Web Security, Network Security, Mobile and Wireless Security;
• Privacy Enhancing Technology, Privacy and Anonymity, Trusted and Trustworthy Computing;
• Risk Evaluation and Security Certification, Critical Infrastructure Protection;
• Security Protocols and Intelligence, Intrusion Detection and Prevention;
• Multimedia Security, Software Security, System Security, Trust Model and Management;
• Security, Privacy, and Trust in Cloud Environments, Mobile Systems, Social Networks, Peer-to-
Peer Systems, Pervasive/Ubiquitous Computing, Data Outsourcing, and Crowdsourcing, etc.

PUBLISHED TITLES

Empirical Research for Software Security: Foundations and Experience

Lotfi ben Othmane, Martin Gilje Jaatun, Edgar Weippl

Intrusion Detection and Prevention for Mobile Ecosystems

Georgios Kambourakis, Asaf Shabtai, Constantinos Kolias, and Dimitrios Damopoulos

Touchless Fingerprint Biometrics

Ruggero Donida Labati, Vincenzo Piuri, and Fabio Scotti

Empirical Research for Software Security: Foundations and Experience

Lotfi ben Othmane, Martin Gilje Jaatun, and Edgar Weippl

Real-World Electronic Voting: Design, Analysis and Deployment

Feng Hao and Peter Y. A. Ryan

Protecting Mobile Networks and Devices: Challenges and Solutions

Weizhi Meng, Xiapu Luo, Steven Furnell, and Jianying Zhou

Location Privacy in Wireless Sensor Networks

Ruben Rios, Javier Lopez, and Jorge Cuellar
 Empirical Research
for Software Security
Foundations and Experience

Edited by Lotfi ben Othmane

Martin Gilje Jaatun • Edgar Weippl
CRC Press
Taylor & Francis Group
6000 Broken Sound Parkway NW, Suite 300
Boca Raton, FL 33487-2742

© 2018 by Taylor & Francis Group, LLC

CRC Press is an imprint of Taylor & Francis Group, an Informa business

No claim to original U.S. Government works

Printed on acid-free paper

Version Date: 20171024

International Standard Book Number-13: 978-1-4987-7641-7 (Hardback)

This book contains information obtained from authentic and highly regarded sources. Reasonable
efforts have been made to publish reliable data and information, but the author and publisher cannot
assume responsibility for the validity of all materials or the consequences of their use. The authors and
publishers have attempted to trace the copyright holders of all material reproduced in this publication
and apologize to copyright holders if permission to publish in this form has not been obtained. If any
copyright material has not been acknowledged please write and let us know so we may rectify in any
future reprint.

Except as permitted under U.S. Copyright Law, no part of this book may be reprinted, reproduced,
transmitted, or utilized in any form by any electronic, mechanical, or other means, now known or
hereafter invented, including photocopying, microfilming, and recording, or in any information
storage or retrieval system, without written permission from the publishers.

For permission to photocopy or use material electronically from this work, please access
www.copyright.com (http://www.copyright.com/) or contact the Copyright Clearance Center, Inc.
(CCC), 222 Rosewood Drive, Danvers, MA 01923, 978-750-8400. CCC is a not-for-profit organization
that provides licenses and registration for a variety of users. For organizations that have been granted
a photocopy license by the CCC, a separate system of payment has been arranged.

Trademark Notice: Product or corporate names may be trademarks or registered trademarks, and
are used only for identification and explanation without intent to infringe.
Visit the Taylor & Francis Web site at
http://www.taylorandfrancis.com
and the CRC Press Web site at
http://www.crcpress.com
Contents

Preface . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ix

List of Figures . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xiii

List of Tables . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xv

Contributors . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . xix

1 Empirical Research on Security and Privacy by Design . . . . . . . . 1

Koen Yskout, Kim Wuyts, Dimitri Van Landuyt, Riccardo Scandariato,
and Wouter Joosen
1.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2
1.2 Empirical Research on Security and Privacy by Design . . . . . . . 5
1.3 Scoping . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9
1.4 Planning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12
1.5 Operation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 30
1.6 Analysis and Interpretation . . . . . . . . . . . . . . . . . . . . . . 35
1.7 Presentation and Packaging . . . . . . . . . . . . . . . . . . . . . 41
1.8 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 43

2 Guidelines for Systematic Mapping Studies in Security Engineering . 47

Michael Felderer and Jeffrey C. Carver
2.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 48
2.2 Background on Systematic Mapping Studies in Software Engineer-
ing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 49
2.3 Overview of Available Mapping Studies in Security Engineering . . 55
2.4 Guidelines for Systematic Mapping Studies in Security Engineering 57
2.5 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 65

v
vi  Contents

3 An Introduction to Data Analytics for Software Security . . . . . . . . 69

Lotfi ben Othmane, Achim D. Brucker, Stanislav Dashevskyi,
and Peter Tsalovski
3.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 70
3.2 Secure Software Development . . . . . . . . . . . . . . . . . . . . 71
3.3 Software Security Analytical Process . . . . . . . . . . . . . . . . 74
3.4 Learning Methods Used in Software Security . . . . . . . . . . . . 82
3.5 Evaluation of Model Performance . . . . . . . . . . . . . . . . . . 86
3.6 More Lessons Learned . . . . . . . . . . . . . . . . . . . . . . . . 89
3.7 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 89
3.8 Acknowledgment . . . . . . . . . . . . . . . . . . . . . . . . . . . 90

4 Generating Software Security Knowledge Through Empirical Methods 95

René Noël, Santiago Matalonga, Gilberto Pedraza, Hernán Astudillo,
and Eduardo B. Fernandez
4.1 Introduction and Motivation . . . . . . . . . . . . . . . . . . . . . 96
4.2 Empirical Methods for Knowledge Generation . . . . . . . . . . . 97
4.3 Example Application Domain: Secure Software Development
Research Project . . . . . . . . . . . . . . . . . . . . . . . . . . . 99
4.4 Experiments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 100
4.5 Systematic Literature Mappings . . . . . . . . . . . . . . . . . . . 112
4.6 Case Studies . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 122
4.7 Experimental Replications . . . . . . . . . . . . . . . . . . . . . . 128
4.8 Conclusions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 132
4.9 Acknowledgment . . . . . . . . . . . . . . . . . . . . . . . . . . . 134

5 Visual Analytics: Foundations and Experiences in Malware Analysis . 139

Markus Wagner, Dominik Sacha, Alexander Rind, Fabian Fischer, Robert
Luh, Sebastian Schrittwieser, Daniel A. Keim, and Wolfgang Aigner
5.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 140
5.2 Background in Malware Analysis . . . . . . . . . . . . . . . . . . 140
5.3 Visual Analytics Foundations . . . . . . . . . . . . . . . . . . . . 143
5.4 The Knowledge Generation Process . . . . . . . . . . . . . . . . . 150
5.5 Design and Evaluation for Visual Analytics Systems . . . . . . . . 152
5.6 Experience in Malware Analysis . . . . . . . . . . . . . . . . . . . 154
5.7 Future Directions . . . . . . . . . . . . . . . . . . . . . . . . . . . 161
5.8 Conclusions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 164

6 Analysis of Metrics for Classification Accuracy in Intrusion Detection 173

Natalia Stakhanova and Alvaro A. Cardenas
6.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 174
6.2 Evaluation Metrics . . . . . . . . . . . . . . . . . . . . . . . . . . 175
6.3 Literature Review . . . . . . . . . . . . . . . . . . . . . . . . . . . 185
6.4 What Hinders Adoption of Alternative Metrics . . . . . . . . . . . 191
6.5 Guidelines for Introducing New Evaluation Metrics . . . . . . . . . 194
 Contents  vii

6.6 Conclusions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 195

6.7 Acknowledgement . . . . . . . . . . . . . . . . . . . . . . . . . . 196

7 The Building Security in Maturity Model as a Research Tool . . . . . 201

Martin Gilje Jaatun
7.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 201
7.2 Background . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 202
7.3 Questionnaires in Software Security . . . . . . . . . . . . . . . . . 202
7.4 A Case Study . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 204
7.5 Discussion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 205
7.6 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 207

8 Agile Test Automation for Web Applications — A Security Perspective 209

Sandra Domenique Ringmann and Hanno Langweg
8.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 210
8.2 Methodology . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 211
8.3 Risk Assessment . . . . . . . . . . . . . . . . . . . . . . . . . . . 212
8.4 Testing and Test Automation from the Security Perspective . . . . . 217
8.5 Static Analysis Tools . . . . . . . . . . . . . . . . . . . . . . . . . 222
8.6 Dynamic Analysis Tools and Frameworks . . . . . . . . . . . . . . 229
8.7 Evaluating Static/Dynamic Analysis Tools and Frameworks . . . . 238
8.8 Appraisal of the Tools . . . . . . . . . . . . . . . . . . . . . . . . 239
8.9 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 240

9 Benchmark for Empirical Evaluation of Web Application Anomaly

Detectors . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 249
Robert Bronte, Hossain Shahriar, and Hisham Haddad
9.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 250
9.2 Literature Review . . . . . . . . . . . . . . . . . . . . . . . . . . . 251
9.3 Benchmark Characteristics for Application-Layer Attack Detection
Approaches . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 256
9.4 An Example Environment for Generating Benchmark Data . . . . . 261
9.5 Using the Benchmark Dataset to Evaluate an IDS . . . . . . . . . . 265
9.6 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 271

10 Threats to Validity in Empirical Software Security Research . . . . . 275

Daniela S. Cruzes and Lotfi ben Othmane
10.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 276
10.2 Defining Validity . . . . . . . . . . . . . . . . . . . . . . . . . . . 277
10.3 Validity for Quantitative Research . . . . . . . . . . . . . . . . . . 278
10.4 Threats to Validity for Qualitative Research . . . . . . . . . . . . . 289
10.5 Summary and Conclusions . . . . . . . . . . . . . . . . . . . . . . 297

Index . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 301
Preface

The software security field has been plagued by “accepted truths” or “self-evident
statements” that at their core are based on nothing more than that some “guru” at one
point thought it sounded like a good idea. Consequently, these “accepted truths” have
often proved to be of varying longevity, as fashion changes and new fads emerge. Em-
pirical research allows to test theories in the real world, and to explore relationships,
prove theoretical concepts, evaluate models, assess tools and techniques, and estab-
lish quality benchmarks across organizations. The methods for doing such research
have been used in several areas, such as social sciences, education, and software en-
gineering. These methods are currently being used to investigate software security
challenges and mature the subject.
The purpose of this book is to introduce students, practitioners and researchers
to the use of empirical methods in software security research. It explains different
methods of both primary and secondary empirical research, ranging from surveys
and experiments to systematic literature mapping, and provides practical examples.
Rather than a complete textbook on empirical research, this book is intended as a
reference work that both explains research methods and shows how software security
researchers use empirical methods in their work. With some chapters structured as
step-by-step instructions for empirical research and others presenting results of said
research, we hope this book will be interesting to a wide range of readers.
In the first chapter, Koen Yskout et al. offer a primer on empirical research in
the area of security and privacy by design, explaining what to expect and what not to
expect as researcher or reviewer. They address the frequent lack of empirical research
on new methods or techniques in the early stages of security and privacy design. Their
experience-led chapter discusses how to design and perform controlled experiments
and descriptive studies in this domain. It contrasts the methods typically applied
by beginning and more experienced researchers with those frequently expected by
reviewers, and strikes a balance between scientific rigor and pragmatism dictated by
the realities of research.
The structured approach guides the reader through the phases of study design,
from research questions and study design to execution to data analysis and dissemi-

ix
x  Empirical Research for Software Security: Foundations and Experience

nation. In many cases, recommendations for additional reading are provided for the
reader seeking to explore a given area more in depth. It not only provides practical
advice for researchers on issues such as using students as test subjects but also in-
cludes helpful tips for reviewers, explaining what to look for in an empirical study
and which allowances to make when reviewing small-scale studies. With this two-
fold approach, it will certainly prove helpful both for empirical researchers who are
just starting out and for reviewers of empirical studies who may not have performed
such studies themselves.
Moving from primary to secondary studies, “Guidelines for systematic mapping
studies in security engineering” by Michael Felderer and Jeffrey C. Carver explains
how to use the systematic mapping method to provide an overview of a research do-
main and to determine which topics have already been extensively covered and which
are in need of additional research. The authors of this chapter illustrate the useful-
ness of systematic mapping studies and provide an overview of systematic mapping
studies previously published in the security engineering field. They compare differ-
ent guidelines for such studies in software engineering and then adapt them to the
security engineering field. Illustrated by examples from actual studies, they provide
extensive methodological support for researchers wishing to conduct such a study,
explaining how to search for and select which studies to include, how to assess their
quality and how to extract and classify the data.
In the following chapter “An Introduction to Data Analytics for Software Secu-
rity,” ben Othmane et al. share their experience on using data analytics techniques to
derive models related to software security at SAP SE, the largest European software
vendor. They use data analytics to study raw data with the purpose of drawing conclu-
sion using machine learning methods or statistical learning methods. They describe
in the chapter the data analytics process that the authors practiced with and give an
overview of a set of machine learning algorithms commonly used in the domain.
They also describe how to measure the performance of these algorithms.
“Generating software security knowledge through empirical methods” by René
No´’el et al. combines both primary and secondary research. The authors explain
how to use experimental methods to generate and validate knowledge about software
security. In addition to a general discussion of validity in research and the use of
empirical methods, they guide the reader step by step through an experimental study,
explaining why the various methods are chosen and what knowledge can be gained
from them. In each section, the theory or method is supplemented with the actual
data from the study. Budding empirical researchers will surely find the explanations
of how to formulate and test a research hypothesis useful. Following the description
of the randomized experiment, the authors explain how they supplemented it with
a systematic literature mapping study and a case study, again detailing the reasons
for and outcomes of each method applied. Another emphasis of this chapter is on the
importance of experimental replication, explaining not just why and how replications
should be conducted but also detailing different types of replications.
The chapter “Visual Analytics: Foundations and Experiences in Malware Analy-
sis by Markus Wagner et al. shows how visual analytics, which combines automated
 Preface  xi

with human analysis by providing powerful visual interfaces for analysts to examine,
can be used to analyze the enormous data loads of malware analysis.
It explains the basics of visual analytics (data processing, models, different vi-
sualization techniques and human interaction with the visualized data, knowledge
generation, and how to design and evaluate visual analytics systems) and how its
methods can be applied to behavior-based malware analysis. This is illustrated with
three projects that used visual analytics for malware analysis. The methods employed
in these projects are compared and used as a basis for recommendations for future
research.
In “Evaluating Classification Accuracy in Intrusion Detection,” Natalia
Stakhanova and Alvaro A. Cárdenas offer an excellent example of how a systematic
literature review can be used to analyze methods employed by the research com-
munity and detect previously unknown ontological issues. They analyze the use of
different evaluation methods for intrusion detection systems (IDSs) and investigate
which factors contribute to or hamper the adoption of new IDS evaluation methods.
They found that the vast majority of researchers use traditional metrics, including
methods that have been criticized as insufficient, and are reticent toward adopting
new ones. In their analysis, they also found a wide variety of different names for the
same metrics, prompting the call for a unified terminology. They also propose guide-
lines for researchers introducing new evaluation metrics, suggesting that new metrics
introduced be explained clearly so that they might be adopted by other researchers as
well. In addition to the literature analysis, this paper discusses the benefits and chal-
lenges of all metrics, and compares IDSs by classification accuracy, and proposes a
framework for the validation of metrics.
Martin Gilje Jaatun explains how the Building Security in Maturity Model
(BSIMM) might be used as an academic research tool. Initially developed by soft-
ware security company Cigital to assess the security maturity level of their clients
by quantifying their software security activities, the BSIMM survey in its original
form was administered in person by representatives of Cigital. It measures twelve
practices in the domains of governance, intelligence, SSDL touchpoints, and deploy-
ment. Jaatun describes how it was converted into a questionnaire with a follow-up
interview. While this method does not provide a BSIMM score in the traditional
sense, the low-threshold approach can yield interesting data for researchers in the
security domain.
In “Agile test automation for web applications,” Sandra Ringmann and Hanno
Langweg address the topic of test automation for security testing. They advocate
the integration of (automated) security testing into the other testing processes of the
software development life cycle. In this very practice-oriented paper, the authors dis-
cuss the main requirements for tools used in agile testing, where testing is performed
by all members of the agile development team, many of whom are not security ex-
perts: they must be user-friendly and human-readable. In addition to a discussion of
different risk rating methodologies and threat models, they provide a thorough and
well-structured overview of different testing methods and tools and explain how to
choose the right one for the job.
xii  Empirical Research for Software Security: Foundations and Experience

The paper presents a number of vulnerability scanners some of which are par-
tially or completely open source and compares their scan results. It also provides an
overview of freely available dynamic analysis tools and presents their use in BDD
(behavior-driven development) frameworks, which allow everyone in the develop-
ment team to participate in or follow the testing process.
In “Benchmark for Empirical Evaluation of Web Application Anomaly Detec-
tors,” Robert Bronte et al. argue the need for a common benchmark for detecting
application-layer attacks. Their chapter provides an overview of benchmarks in the
field previously suggested by other researchers and compares their advantages and
disadvantages as well as the attributes they focused on before setting out to define
the characteristics a unifying benchmark for application-layer attack detection would
have to have. They pay careful attention to the environment required to generate
benchmark data and demonstrate how such data could be used to evaluate an intru-
sion detection system.
Validity is the extent to which the design and conduct of empirical studies are
likely to prevent systematic errors or bias. Empirical research studies are associated
always with validity threats that limit the use of the results. Cruzes and ben Othmane
provide in the chapter “Threats to Validity in Software Security Empirical Research”
a taxonomy of validity threats that apply to secure software engineering qualitative
and quantitative studies. In addition, they give examples on how these threats have
been addressed or discussed in the literature. Rigorous threats to validity helps to
advance the common knowledge on secure software engineering.
The back cover picture is provided by Srdjan Pavelic.
We hope that this book provides an interesting introduction into the use of em-
pirical research methods and helps researchers and practitioners alike select the ap-
propriate evaluation method for their project.
List of Figures

1.1 The custom tool implemented for the SPAT2 study guides the
participant through all the required steps. . . . . . . . . . . . . . . . 28

2.1 Process Steps for Systematic Mapping Studies. . . . . . . . . . . . 50

2.2 Classification Scheme for Model-Based Security Testing. . . . . . . 64

3.1 Overview of the S2 DL . . . . . . . . . . . . . . . . . . . . . . . . . 71

3.2 Software security analytical process. . . . . . . . . . . . . . . . . . 74
3.3 Plot that visualizes missing data. . . . . . . . . . . . . . . . . . . . 78
3.4 Duration by vulnerability. . . . . . . . . . . . . . . . . . . . . . . . 80

4.1 Running Example and Empirical Methods. . . . . . . . . . . . . . . 101

4.2 Box plot for effort and number of threats (QoT) variables. . . . . . . 109
4.3 Box plot for Experience groups . . . . . . . . . . . . . . . . . . . . 110
4.4 Studies distribution plot . . . . . . . . . . . . . . . . . . . . . . . . 120
4.5 Quantitative data analysis. . . . . . . . . . . . . . . . . . . . . . . 126

5.1 Components of a VA system: data, model, and visualization. . . . . 145

5.2 Sample images of the five visualization categories defined by Keim. 147
5.3 The knowledge generation model for visual analytics combines
human and machine concepts. . . . . . . . . . . . . . . . . . . . . 150
5.4 Nested model for visualization design and evaluation by Munzner. . 153
5.5 MalwareVis [24] interactively visualizes network activity of an indi-
vidual malware sample. . . . . . . . . . . . . . . . . . . . . . . . . 155
5.6 SEEM [22] : The system, which is built into Cynomix, provides a vi-
sual environment to support feature-based malware comparison for
large attribute sets of malware samples. The highly interactive web-
based system provides an overview about features of malware sam-
ples compared to other related or similar malware samples or fami-
lies. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 157
xiii
xiv  Empirical Research for Software Security: Foundations and Experience

5.7 KAMAS [65]: a knowledge-assisted visualization system for

behavior-based malware analysis. Image by the authors. . . . . . . . 159
5.8 A comparison of the three tools based on the knowledge generation
model. Interactive components are illustrated with dashed borders.
The strength of the loops (color and border) denote how well the
loops are supported by the tools. Image by the authors. . . . . . . . 160

6.1 Authentication protocol V should output 1 if and only if P is who she

claims to be. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 175
6.2 (a) Evaluation of the correctness and security of an authentication
algorithm. (b) Our problem: we can no longer achieve negligible er-
ror probabilities, i.e. a cannot be made as small as possible without
increasing b. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 176
6.3 B-ROC curve. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 184
6.4 The summary of the reviewed publications. . . . . . . . . . . . . . 187
6.5 The use of evaluation metrics over time. . . . . . . . . . . . . . . . 188
6.6 The use of evaluation metrics among research groups and among the
conferences. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 190

7.1 Conservative maturity for the three most mature organizations . . . 205

8.1 OWASP Top 10 application security risks. . . . . . . . . . . . . . . 213

8.2 The cycle of continuous integration adapted from. . . . . . . . . . . 219
8.3 YASCA’s architecture. . . . . . . . . . . . . . . . . . . . . . . . . 227

9.1 An Apache2 Attack Illustration . . . . . . . . . . . . . . . . . . . . 257

9.2 User-to-Root Attack Diagram . . . . . . . . . . . . . . . . . . . . . 258
9.3 The Environment for Data Generation . . . . . . . . . . . . . . . . 263
9.4 Content Management System Log Entries . . . . . . . . . . . . . . 264
9.5 Blogging Platform Log Entries . . . . . . . . . . . . . . . . . . . . 264
9.6 Bulletin Board System Log Entries . . . . . . . . . . . . . . . . . . 264
9.7 Classifieds Marketplace Log Entries . . . . . . . . . . . . . . . . . 264
9.8 E-commerce Platform Log Entries . . . . . . . . . . . . . . . . . . 264
9.9 Comparison between CEP, length and MD measures . . . . . . . . 267
9.10 Comparison between CEV, length and MD measures . . . . . . . . 267
9.11 Comparison between CET, length and MD measures . . . . . . . . 267
9.12 Information-theoretic IDS Framework . . . . . . . . . . . . . . . . 270
9.13 Log files created by web applications deployed in Apache and stored
in MySQL . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 271
List of Tables

1.1 An illustration of typical discrepancies between beginning

researchers, experienced researchers, and reviewers. . . . . . . . . . 4
1.2 Empirical research for security by design: running examples . . . . 7
1.3 Empirical research for privacy by design: running examples . . . . . 8

2.1 Guidelines used for Systematic Mapping Studies in Security

Engineering . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 54
2.2 Security Engineering Mapping Studies . . . . . . . . . . . . . . . . 55
2.3 Number of Retrieved and Included Papers in Available Mapping
Studies in Security Engineering. . . . . . . . . . . . . . . . . . . . 56
2.4 Main Security Journals Covering Software Security Engineering
Topics. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 58
2.5 Main Security-Related Conferences Covering Software Security
Engineering Topics. . . . . . . . . . . . . . . . . . . . . . . . . . . 58
2.6 Main Software Engineering Venues Covered by Security
Engineering Mapping Studies. . . . . . . . . . . . . . . . . . . . . 59
2.7 Common Databases Used for Software Security Engineering
Mapping Studies . . . . . . . . . . . . . . . . . . . . . . . . . . . 60

3.1 Selected set of machine learning methods. . . . . . . . . . . . . . . 83

3.2 Confusion matrix for two-class outcome variables. . . . . . . . . . 86

4.1 Levels of evidence, extended from Kitchenham et al. . . . . . . . . 97

4.2 Taxonomy of empirical methods . . . . . . . . . . . . . . . . . . . 98
4.3 Attainable evidence by method . . . . . . . . . . . . . . . . . . . . 98
4.4 Scoping template adapted from . . . . . . . . . . . . . . . . . . . . 102
4.5 Recommended descriptive statistics. . . . . . . . . . . . . . . . . . 104
4.6 Scoping for the original experiment . . . . . . . . . . . . . . . . . 105
4.7 Experiment Design . . . . . . . . . . . . . . . . . . . . . . . . . . 107
4.8 Experiment instruments . . . . . . . . . . . . . . . . . . . . . . . . 107
xv
xvi  Empirical Research for Software Security: Foundations and Experience

4.9 Experiment Execution . . . . . . . . . . . . . . . . . . . . . . . . 108

4.10 Medians for Experience groups . . . . . . . . . . . . . . . . . . . . 110
4.11 Search string definition . . . . . . . . . . . . . . . . . . . . . . . . 116
4.12 Source selection . . . . . . . . . . . . . . . . . . . . . . . . . . . . 117
4.13 Inclusion/Exclusion Criteria . . . . . . . . . . . . . . . . . . . . . 118
4.14 Studies selection procedure . . . . . . . . . . . . . . . . . . . . . . 119
4.15 Selected papers from first filtering . . . . . . . . . . . . . . . . . . 120
4.16 Resume of number of papers selected in each phase . . . . . . . . . 120
4.17 Search string definition . . . . . . . . . . . . . . . . . . . . . . . . 121
4.18 Qualitative Analysis: Constant Comparison Method application . . . 127
4.19 The family of experiments: Goal and Context . . . . . . . . . . . . 133

6.1 True positives (TP) are known attack instances detected as abnormal,
false positives (FP) are normal instances that are incorrectly classi-
fied as abnormal, true negatives (TN) are normal instances correctly
identified as normal behavior, and false negatives (FN) present ab-
normal behavior incorrectly classified as normal. . . . . . . . . . . 177
6.2 Traditional metrics. . . . . . . . . . . . . . . . . . . . . . . . . . . 178
6.3 Summary of symbols. . . . . . . . . . . . . . . . . . . . . . . . . . 181
6.4 Costs of IDS reports given a state of a system . . . . . . . . . . . . 182
6.5 Costs model for detector outcome for a given event. DCost(x) ≤
RCost(x) is a degenerate case that will not apply in most cases. . . . 182
6.6 The summary of the reviewed publications. . . . . . . . . . . . . . 186
6.7 The evaluation metrics employed in the reviewed works. . . . . . . 186
6.8 Summary of alternative evaluation metrics. . . . . . . . . . . . . . 191

7.1 The BSIMM Software Security Framework . . . . . . . . . . . . . 203

8.1 OWASP Top Ten 2013 . . . . . . . . . . . . . . . . . . . . . . . . 214

8.2 Risk rating scores (adapted from. . . . . . . . . . . . . . . . . . . . 215
8.3 Example risk calculation of A3-Cross-Site Scripting. . . . . . . . . 216
8.4 Likelihood and impact levels adapted from. . . . . . . . . . . . . . 216
8.5 Overall risk severity. . . . . . . . . . . . . . . . . . . . . . . . . . 217
8.6 Overview of static analysis tools referenced in. . . . . . . . . . . . 223
8.7 Overview of dynamic analysis tools, i.e., vulnerability/web scanners
and penetration testing tools. . . . . . . . . . . . . . . . . . . . . . 229
8.8 Overview of dynamic analysis frameworks. . . . . . . . . . . . . . 231

9.1 Summary of detailed Literature Review . . . . . . . . . . . . . . . 254

9.2 Measured Data Attributes from the Literature . . . . . . . . . . . . 259
9.3 Related Works on Anomaly-based IDS . . . . . . . . . . . . . . . . 268

10.1 Threats to conclusion validity in quantitative research. . . . . . . . . 280

10.2 Threats to internal validity in quantitative research. . . . . . . . . . 281
10.3 Threats to construct validity in quantitative research. . . . . . . . . 282
10.4 Threats to external validity in quantitative research. . . . . . . . . . 284
 List of Tables  xvii

10.5 Validity criteria from different authors. . . . . . . . . . . . . . . . . 290

10.6 Techniques for addressing threats to validity in qualitative research. 292
Contributors

Hernán Astudillo Eduardo B. Fernandez

Universidad Técnica Federico Santa Florida Atlantic University
Marı́a Boca Raton, Florida, USA
San Joaquı́n, Santiago, Chile
Hisham Haddad
Robert Bronte Kennesaw State University
Kennesaw State University Kennesaw, Georgia, USA
Kennesaw, Georgia, USA
Martin Gilje Jaatun
Achim D. Brucker SINTEF
The University of Sheffield Trondheim, Norway
Sheffield, UK
Wouter Joosen
Alvaro A. Cardenas imec-DistriNet, KU Leuven
University of Texas at Dallas Leuven, Belgium
Richardson, Texas, USA
Dimitri Van Landuyt
Jeffrey C. Carver imec-DistriNet, KU Leuven
University of Alabama Leuven, Belgium
Tuscaloosa, Alabama, USA
Hanno Langweg
Daniela S. Cruzes HTWG Konstanz University of
SINTEF Applied Sciences
Trondheim, Norway Konstanz, Germany

Stanislav Dashevskyi Santiago Matalonga

University of Trento Universidad ORT Uruguay
Trento, Italy Montevideo, Uruguay

Michael Felderer René Noël

University of Innsbruck Universidad de Valparaı́so
Innsbruck, Austria Valparaı́so, Chile
xix
xx  Empirical Research for Software Security: Foundations and Experience

Lotfi ben Othmane Hossain Shahriar

Fraunhofer Institute for Secure Kennesaw State University
Information Technology Kennesaw, Georgia, USA
Darmstadt, Germany
Peter Tsalovski
Gilberto Pedraza SAP SE
Universidad de Los Andes and Walldorf, Germany
Universidad Piloto de Colombia
Bogotá, Colombia
Edgar Weipp
Sandra Domenique Ringmann SBA Research
Wien, Austria
HTWG Konstanz University of
Applied Sciences
Konstanz, Germany Kim Wuyts
imec-DistriNet, KU Leuven
Riccardo Scandariato Leuven, Belgium
Chalmers & University of
Gothenburg Koen Yskout
Gothenburg, Sweden imec-DistriNet, KU Leuven
Leuven, Belgium
Natalia Stakhanova
University of New Brunswick
Fredericton, Canada
Chapter 1

Empirical Research on
Security and Privacy by
Design
What (not) to expect as a
researcher or a reviewer

Koen Yskout, Kim Wuyts, Dimitri Van Landuyt, Riccardo Scandariato,

and Wouter Joosen

CONTENTS
1.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2
1.2 Empirical Research on Security and Privacy by Design . . . . . . . . . . . . . . 5
1.3 Scoping . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9
1.3.1 Setting Verifiable Goals . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9
1.3.2 Process- or Result-Oriented . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10
1.3.3 Quality or Quantity . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11
1.4 Planning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12
1.4.1 Defining Research Questions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12
1.4.1.1 Determining appropriate measures . . . . . . . . . . . 13
1.4.1.2 Defining success criteria . . . . . . . . . . . . . . . . . . . . . 14

1
2  Empirical Research for Software Security: Foundations and Experience

1.4.1.3 Be prepared for surprises . . . . . . . . . . . . . . . . . . . . . 15

1.4.2 Study Design . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16
1.4.2.1 Open or restricted environment . . . . . . . . . . . . . . . 16
1.4.2.2 Exploring the study potential . . . . . . . . . . . . . . . . . 18
1.4.2.3 Controlling variance . . . . . . . . . . . . . . . . . . . . . . . . . 19
1.4.3 Students as Subjects . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21
1.4.4 Instrumentation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23
1.4.4.1 Study objects . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23
1.4.4.2 Guidelines and other material . . . . . . . . . . . . . . . . 25
1.4.4.3 Measurement instruments . . . . . . . . . . . . . . . . . . . . 26
1.4.5 Evaluating Your Own Proposals . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29
1.5 Operation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 30
1.5.1 Defining a Baseline . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31
1.5.2 Motivating and Training Participants . . . . . . . . . . . . . . . . . . . . . . . 32
1.5.3 Limited Time Frame . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 33
1.6 Analysis and Interpretation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 35
1.6.1 Preparing Data . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 35
1.6.1.1 Anonymizing the dataset . . . . . . . . . . . . . . . . . . . . . 35
1.6.1.2 Data extraction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 36
1.6.1.3 Dealing with outliers . . . . . . . . . . . . . . . . . . . . . . . . . 37
1.6.2 Data Analysis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 38
1.6.3 Interpreting Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 39
1.7 Presentation and Packaging . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41
1.7.1 Replications . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41
1.7.2 Presentation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 42
1.8 Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 43
References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 44

1.1 Introduction
Research on software security and privacy is very active, and new techniques and
methods are proposed frequently. In practice, however, adoption is relatively slow,
especially for techniques and methods in the early software engineering phases (re-
quirements elicitation, architecture and design). Yet it is precisely in these early de-
sign phases that the security-by-design (and privacy-by-design) principles are ex-
pected to yield substantial returns on investment: a little extra early development
effort may avoid lots of late re-engineering efforts.
Although these arguments are intuitively convincing, it is our belief that a lack of
empirical evidence to support claims about the benefits of early security design is one
of the main impediments to adoption. Empirical research is an essential technique to
study whether a new method or technique has the promised effects, but also to val-
idate whether it is feasible in practice. Despite their importance, such studies are
not performed as often as they should be — there are many hurdles and roadblocks,
especially for privacy and security engineering! Quantifying the level of security or
 Empirical Research on Security and Privacy by Design  3

privacy of a design is far from trivial. In addition, an attacker can use several ap-
proaches to breach a system. Determining whether a security objective was met, is
thus challenging. Also, given its sensitive nature, obtaining the security documenta-
tion of an industrial software architecture is hard and therefore performing realistic
studies can be difficult.
In this chapter, we share our experiences from the past five years with perform-
ing empirical studies specifically related to early security and privacy design activ-
ities (e.g., [27, 25, 35, 38, 37, 3, 24, 4]). Our empirical research experience mainly
consists of controlled experiments and descriptive studies. We present approaches to
perform such studies that have worked for us, discuss challenges that we have en-
countered along the way, and present remedies that we have effectuated to address
these challenges. We provide experience-driven recommendations both for (begin-
ning) empirical researchers, as well as for reviewers of empirical studies.
To sketch the context, table 1.1 (in the first and second column) illustrates (in
a caricatural manner) some of the typical discrepancies that exist between how be-
ginning researchers — or, for that matter, researchers performing pilot validations of
their approach — undertake such studies, in comparison to more experienced empiri-
cal researchers. The second versus the third column also exemplifies the mismatch in
expectations about empirical studies from the point of view of researchers on the one
hand, and external stakeholders such as scientific reviewers on the other. The latter
discrepancy is often, at least in part, caused by different expectations with respect to
the internal and external validity of the study, and has already been described well by
others (e.g., Siegmund et al. [28]).
In this chapter, we share our practical experiences with performing such studies.
We provide concrete attention points for various aspects of designing, organizing,
executing, processing and publishing about empirical studies. This chapter is there-
fore particularly useful for security researchers interested in conducting empirical
research, as well as for scientific reviewers, as it gives some concrete pointers for
assessing such studies. The chapter is in part anecdotal, by referring to concrete inci-
dents and by citing reviewer comments that we have received. This necessarily takes
these situations and quotes out of their context, at the risk of losing some of the
nuances that were originally present.
The common theme throughout the chapter is what makes these experiments
highly challenging: empirical security researchers are continually forced to make dif-
ficult trade-offs between, on the one hand, the scientific rigor essential in scientific
and empirical experimentation, and on the other hand, the required level of pragma-
tism to make such studies happen, especially when they involve human participants.
The chapter is structured according to the main phases of the process for con-
ducting an empirical study, where we adopt the terminology used by Wohlin et al.
[33]: scoping (Section 1.3), where the overall goals and objectives of the study are
defined; planning (Section 1.4), which involves the careful design of the study; op-
eration (Section 1.5), focusing on preparation of subjects, and the actual execution
to collect data; analysis and interpretation (Section 1.6), i.e., exploring and sani-
tizing the data, and making scientifically sound conclusions; and presentation and
packaging (Section 1.7), where the conclusions about the data and research materials
4  Empirical Research for Software Security: Foundations and Experience

Table 1.1: An illustration of typical discrepancies between beginning researchers,

experienced researchers, and reviewers.

To assess the value of a security- or privacy-by-design approach, . . .

a beginning researcher an experienced re- a reviewer expects
performs searcher performs

B a small validation ex- B a controlled experi- B several replicated con-

ercise, ment, trolled experiments,

B involving a handful of B involving N represen- B involving at least 100 ∗

peers, tative participants, N experienced industrial
developers, experts and
practitioners,

B who solve a small ex- B who solve a well- B who perform a large-
ample problem, scoped design exercise, scale industrial develop-
ment project,

B after receiving a short, B after receiving a B after having several

ad-hoc introduction to structured tutorial months of practical expe-
the approach, about the approach, rience in applying the ap-
proach,

B where the approach in- B where the approach B where the approach
troduces a specific secu- supports a specific secu- supports the entire devel-
rity or privacy design ac- rity or privacy design ac- opment life-cycle,
tivity technique, tivity,

B resulting in measures B resulting in the quan- B resulting in the quan-

that are determined only titative evaluation of a tification of the influence
after the data was col- specific, well-defined hy- of the approach on se-
lected and processed. pothesis concerning the curity, privacy, produc-
approach. tivity, compatibility with
existing industrial prac-
tices, . . . .

are wrapped up for publication. We do not intend to explain or even touch upon ev-
ery activity in this process; elaborate descriptions can be found elsewhere (e.g., [33,
Chapters 6–11]). But first, the following section sketches the context of our studies
that are used as an example throughout the chapter.
 Empirical Research on Security and Privacy by Design  5

1.2 Empirical Research on Security and Privacy

by Design
There is an increasing awareness both in industry and in academic research that
complex non-functional cross-cutting concerns such as security and privacy inher-
ently require up-front attention, much in line with the principles of (software) quality
by design. One example is the recent update of EU regulations which stipulate that
software-intensive systems and services involving the processing of user data should
be designed according to privacy-by-design principles [11]. In turn, many security
vulnerabilities, bugs, and leaks find their roots at the level of the software architec-
ture, because software is built with specific assumptions in mind, which — when
invalidated by attackers — cause breakage of such systems (see [1] and [6], for ex-
ample).
As part of our empirical research, we have studied a number of security and
privacy by design methods, notations, and techniques that focus on the early stages of
the software development and that aim at bringing the security and privacy by design
principles into practice. From the many available techniques (see the surveys in [3, 7,
20, 30], for example), our efforts have focused primarily on STRIDE [15] for security
threat elicitation and mitigation, LINDDUN [9, 34] as its counterpart for privacy, and
architectural security patterns [36, 26], and security modeling notations [3].
A key question is whether this early development effort really pays off. Are these
currently-existing security and privacy design techniques capable of identifying po-
tential issues before they turn into actual problems, and do they effectively lead to
software designs that are inherently less prone to security and privacy defects? These
are by no means trivial questions to answer, and combined with the observation that
empirical studies about these questions are performed far too infrequently, we con-
clude that empirical evidence is lacking to support such claims.
This book chapter crystallizes our main lessons learned, do’s and don’ts, tips and
tricks, and shares some of our experiences and war stories from over five years of
empirical research on security and privacy in the early stages of the software devel-
opment life cycle (requirements elicitation and analysis, and architectural design).
Tables 1.2 and 1.3 below summarize our track record in conducting empirical studies
on respectively security by design and privacy by design. We draw examples from
these studies throughout the chapter, using the acronyms given in the top row of the
tables when referring to them.
In the remainder of this section, we briefly sketch the purpose of each of the stud-
ies, which may help to better understand the examples in this chapter. Note that the
rest of this chapter intentionally does not discuss the topic or even the findings of our
studies, but focuses exclusively on the aspects related to their planning and execu-
tion. We gladly refer the interested reader to the corresponding research publications
for more information about the results.
In the Security threat modeling (STM) [27] study, we have investigated the cost
and effectiveness of Microsoft’s STRIDE [15] by assessing its correctness, com-
pleteness, and productivity. The study concluded that STRIDE is relatively time-
6  Empirical Research for Software Security: Foundations and Experience

consuming to execute, but fairly easy to learn and execute. Nevertheless, we have
observed that many threats remained undetected during the analysis.
In the Security threat modeling (STM2) [unpublished] study, we are inves-
tigating how the correctness and completeness of a security threat analysis using
STRIDE is affected by the level of detail in which the data flows of the system are
described. The threats elicited by the participants are compared with a baseline that
is independently defined by experts.
In the Secure architectural design with patterns (SPAT1) [38] study, we have
investigated whether providing a fine-grained, systematic structure on top of a cata-
log of security patterns (as suggested by multiple researchers in the field) improves
the performance of the software designer in terms of overall time spent, and the ef-
ficiency of finding and selecting a pattern. We concluded that adding more structure
can be beneficial, but that this is not self-evident.
In the Secure architectural design with patterns (SPAT2) [37] study, a follow-
up of the SPAT1 study, we have investigated whether the availability of security pat-
terns increases the security of the resulting design and/or the performance of the
designer. The study has lead to the observation that security patterns, in their current
form, do not yet achieve their full potential, and that there exists a need for improving
them.
In the Privacy threat analysis at requirements level (PAR) [35] study, we have
investigated the cost and effectiveness of the LINDDUN privacy threat modeling
framework [34] during the early stages of software development. Give the limited
amount of participants, this study only had an exploratory nature and mainly focused
on retrieving feedback from the participants regarding ease of use.
In the Privacy threat analysis at architectural level (PAA) [35] study, we have
investigated the cost and effectiveness of the LINDDUN privacy threat modeling
framework during the architectural design phase. We observed similar results com-
pared to our STRIDE study STM. Although the completeness rate of LINDDUN
turned out to be even beter than STRIDE’s, we have found that the productivity was
only half.
In the Privacy methodology comparison with privacy experts (PCE) [35]
study, we have investigated the reliability of the LINDDUN privacy threat model-
ing framework, by comparing the analysis results of privacy experts with those of
LINDDUN. We observed that LINDDUN was missing coverage in the areas of data
minimization and data inference, which in turn allowed us to improve the LIND-
DUN methodology. However, LINDDUN did cover a number of threats that were
overlooked by the privacy experts.
In the Privacy threat modeling (PTM) [unpublished] study, the privacy equiv-
alent of the STM2 study mentioned above, we are investigating how the correctness
and completeness of a privacy threat analysis using LINDDUN is affected by the
level of detail in which the data flows of the system are described. Again, the threats
elicited by the participants are compared with a baseline that is independently defined
by experts.
 Empirical Research on Security and Privacy by Design  7

Table 1.2: Empirical research for security by design: running examples

STM STM2 SPAT1 SPAT2
Type of activity

Security Threat Security Threat Secure architec- Secure architec-

Modeling [27] Modeling [un- tural design with tural design with
published] Patterns [38] Patterns [37]
Study goals

Productivity, Correctness, com- Performance and Security (sound-

correctness, pleteness, produc- efficiency, impact ness and com-
completeness tivity of pattern catalog pleteness of
structure solutions), perfor-
mance
Number of participants

10 teams (41 mas- 93 participants 45 teams (90 mas- 32 teams (64 mas-
ter students) (master students) ter students) ter students)
Quantitative vs. qualitative

Quantitative Quantitative, with Quantitative (and Quantitative (and

exploration of some qualitative) some qualitative)
ease of use
Output

Templated threat Templated threat Report, tool Report, tool

report report, question- measurements measurements
naires (pattern cata- (secured design
log browsing models, time,
history, UML questionnaires)
models, time,
questionnaires)
Environment

Open (10 days of- Restricted (2.5h Mixed (2 super- Mixed (3 super-
fline + 1 lab ses- lab session, no vised lab sessions vised lab sessions
sion) communication, of 2.5h + work at of 2.5h + work at
all on paper) home, at all times home, at all times
restricted to using restricted to using
provided tool) provided tool)
8  Empirical Research for Software Security: Foundations and Experience

Table 1.3: Empirical research for privacy by design: running examples

PAR PAA PCE PTM
Type of activity

Privacy threat Privacy threat Privacy Privacy Threat

Analysis at Analysis at methodology Modeling [un-
Requirements Architectural Comparison with published]
level [35] level [35] privacy Experts
[35]
Study goals

Correctness, com- Correctness, com- Reliability Correctness, im-

pleteness, produc- pleteness, produc- pact of domain
tivity, ease of use tivity, ease of use knowledge
Number of participants

3 teams (8 profes- 27 teams (54 mas- 5 participants (2 122 participants

sionals) ter students) methodology ex- (master students)
perts + 3 privacy
experts)
Quantitative vs. qualitative

Qualitative, with Quantitative, with Quantitative Quantitative, with

exploration of exploration of exploration of
quantitative goals ease of use ease of use
Data

Templated threat Templated threat Templated threat Templated threat

report, question- report, ques- report report, question-
naire, post-it dis- tionnaires, self- naires
cussion session reported time
tracking, access
logs of catalog
Environment

Mixed (13 hours Open (2 weeks Open (Offline, no Restricted (2.5h

of lab sessions di- offline time + time limitation) lab session, no
vided over 3 days. 2 lab sessions. communication,
No limitation of Self-reported all on paper)
technology or time tracking.)
communication.)
Another random document with
no related content on Scribd:
again.’ She then drew back and veiled her face as her father
approached, followed by Embarek and the two prisoners.
Addressing the latter, Sheikh Shashon said, ‘At the intercession of
João, whom I take to-morrow to the Court to enter the service of our
Lord and Master, as gunsmith, your lives are spared and your fetters
shall be removed. You will be taken with João to the Sultan, and
upon His Majesty’s decision your fate will depend. I swear, however,
that if you attempt to escape, no mercy shall be shown you.’
‘Take them,’ he continued to the slave, ‘to your hut and lock them
in; but remove their fetters. Let them have food from my kitchen that
they may feel well and strong for the journey to-morrow. Put a couch
for João in the courtyard: he is my guest, free to come and go as he
pleases.’ Then turning towards Rahma, he said, smiling, ‘All this I do
to please you, my loved daughter.’
‘May God bless her!’ cried João and his companions.
Early on the following morning the Sheikh mounted a fine mule,
and the prisoners the animals prepared for them; whilst, destined as
a present to the Sultan, the famous gray mare, adorned with a
handsome headstall, was led by a slave.
Rahma appeared on the threshold, muffled in her ‘haik’; but
before João left she managed, when her father’s back was turned, to
unveil her face, and drawing from her bosom, where she had hidden
them, the silver chain and cross, pressed them to her lips: which
gesture João acknowledged by raising towards heaven the finger
upon which he wore her ring.
Sheikh Shashon despatched a courier to the Court to announce
their advent, and fearing lest some enemy in the village might
forestall him, he wrote to the Uzir that he was bringing the gunsmith
João and two other Nazarenes, prisoners, to deliver them to his Lord
and Master the Sultan, to be dealt with as His Majesty might please.
When within a few hours’ journey of the capital a Kaid of the
Sultan’s body-guard, sent expressly by His Majesty, arrived with an
order to the Sheikh to the effect that every care should be taken of
João, and to inform the latter that a house and forge, where he could
work, had already been prepared for him, and that the two other
prisoners were to be lodged for the present in the same dwelling.
The Kaid also informed the Sheikh that His Majesty commended his
conduct in having brought João safely to the Court, and that the
Sheikh was therefore regarded favourably by his Lord and Master.
On his arrival João was taken before the Sultan, who informed
him that he would be provided with ‘mona’ (provisions), and a
dwelling near the palace; that the implements of a smith and piles of
old horse-shoes were also ready, and that for every gun-barrel João
made, ten ducats would be paid him. The Sultan added, ‘If you will
become one of the Faithful, I have ordered that the garments of a
Moslem be given you.’
João thanked His Majesty and replied, ‘I accept with pleasure
your Majesty’s offer of Moorish garments to replace the tattered
clothing I now wear.’
Whilst thus accepting the Sultan’s offer, João vowed in his heart
that, though assuming the outward garb of a Mohammedan in the
hope of obtaining Rahma hereafter as his wife, he would remain
always a true Catholic, and hope for the day when he would return to
the land of his forefathers.
João was very industrious, and with the assistance only of the two
Portuguese, his fellow-prisoners—for he did not wish the Moors to
discover the secret of his art—he was enabled to manufacture a
number of barrels, even before the Sheikh left the Court.

The Sultan[44], who was interested in every kind of mechanism,

was wont to go to the forge to see João work; gave him the rank of
Kaid, and marked in many ways his satisfaction.
The Sheikh was presented with a horse, with handsome saddle
and bridle, as a mark of His Majesty’s favour, and before leaving the
Court went to see João, and told him of his own good fortune, and
expressed his satisfaction at seeing from his dress that João was
now a Moslem and an officer in high favour with the Sultan.
João shook the Sheikh warmly by the hand, bidding him farewell,
saying, ‘You know that I am indebted for my life to the intercession of
your daughter. I intend to marry and settle here. Will you grant me
the hand of your daughter?’
‘It cannot be,’ answered the Sheikh, ‘I have betrothed her to my
friend Sheikh Amar. The Sultan, now that you are in such high
favour, will bestow on you, if you petition His Majesty, some maiden
with a larger dowry than I can afford to give my daughter.’ He then
departed, leaving João very depressed.
A few days after the Sheikh had left, the Sultan visited the forge of
João and found the young smith hard at work, but looking very wan
and out of spirits. Observing this, the Sultan inquired of João
whether he was unwell, or had cause of complaint against any one
at the Court, and whether the food sent daily from the palace was
plentiful and such as he liked?
João replied that he had no complaint to make against any one,
but that he had a sorrow at heart which he could not make known to
the Sultan, lest it might cause His Majesty’s displeasure.
 ‘Speak,’ said the Sultan; ‘have no fear. Any one who may have
offended you shall be punished. Whatever you ask shall be granted:
what I promise shall be fulfilled. Speak out boldly.’
João obeyed and told the Sultan the story of his capture,
condemnation to death, and release at the intercession of the
Sheikh’s daughter.
When he had concluded his tale, His Majesty exclaimed, ‘Allah
Akbar!’ (God is great!) ‘Had the Sheikh taken your life he would have
forfeited his own. This daughter of his, the maiden who is the cause
of my having you safe here to manufacture guns for the Moslems,
shall be rewarded. What do you desire?’
Throwing himself at the Sultan’s feet João said, ‘She who saved
my life I had hoped might become my wife, but alas! I have learnt
she is betrothed to a friend of the Sheikh, an old chief of a
neighbouring village, named Sheikh Amar. This it is that makes me
miserable.’
‘Before ten days elapse,’ said the Sultan, ‘if this maiden be not
already married to Sheikh Amar, she shall be brought here by her
father and become your wife, and I will give her a dowry.’
The young smith again fell at the feet of the Sultan and expressed
his gratitude.
A Kaid was despatched with all speed to the Sheikh of Beni
M’suar, with the command that he and all his family should be
brought at once to the Court. This officer was directed however to
ascertain, before he executed this order, whether the daughter of the
Sheikh had been lately married; for in such case the Royal command
was not to be carried out.
 The officer departed on his mission and found that the wedding
had not taken place, as old Sheikh Amar had died suddenly shortly
after Sheikh Shashon had left for Fas. Father and daughter were
therefore brought to the Court, and on their arrival were given a
comfortable dwelling near the palace.
Rahma’s heart was filled with joy when she learnt that João was
in high favour with the Sultan, for she remembered his last words to
herself.
The smith hastened to salute the Sheikh. Rahma was not allowed
to enter the room, but she could see her lover through the chinks of
the door, and heard João, after saluting her father, say, ‘Is your
daughter, who saved my life, well? Is she unmarried? If so, I must
not conceal from you that I have petitioned the Sultan that she be
given me as wife. For this His Majesty has been pleased to order
you to come to the Court.’
The Sheikh, who had been in great trepidation, fearing that the
Sultan might have heard of the intention he at one time had of
putting João and the other Portuguese to death, and that His Majesty
had summoned him to the Court to punish him, was greatly relieved,
and replied,—
‘Oh my son! as your garb shows you are now one of the Faithful
and in favour with our Lord and Master, His Majesty’s commands,
whatever they may be, shall be joyfully obeyed.’
The Sultan ordered the Uzir to signify to the Sheikh his Royal
command that his daughter was forthwith to be wedded to João, and
that it was His Majesty’s intention to give her a handsome dowry.
A great feast was prepared by the officers of the Court, at which
the Sheikh attended, whilst Rahma was taken to the harem of the
Hajib (Chief Chamberlain), where the ladies had also prepared a
feast. Beautiful dresses and jewelry were sent by the Sultan to
Rahma, and a marriage contract was drawn up by public notaries,
signed by the Kadi, with a note of the dowry, one thousand ducats,
given her by the Sultan.
On the day of the wedding, the bride, ensconced in a wooden
cage, covered with silk and embroidery, was conveyed on the back
of a mule to João’s house, accompanied by musicians with pipes
and drums and a large troop of men firing guns. The cage was
removed from the back of the mule by four female slaves and
brought into the room, prepared with handsome carpets, where João
awaited her. The slaves assisted her to leave the cage and retired.
As soon as they were alone Rahma threw herself at the feet of
her husband, crying, ‘Oh beloved! God has answered our prayers.
He is merciful, and now I shall be, as long as I live, your faithful,
happy wife. But, João, I beg you to repeat that you believe in God
and the Day of Resurrection. I rejoice to see you in the garb of a
Moslem, and hope you are now really one of the Faithful.’
‘Rahma,’ he said, raising her in his arms, ‘to thee I owe my life; for
thee I shall be ready to lay it down; but I must not deceive thee! I am
not a Moslem, but a Christian, and, as such, I believe in God and the
last Day. I assumed this garb in order that I might be supposed to be
a Mohammedan, and thus be able to petition the Sultan that you
should be my wife.’
Rahma drew away from his arms, saying, ‘I cannot, I must not,
offend God by marrying a Christian.’
João replied, ‘Know you not that your prophet Mohammed
married a Christian woman? Oh loved wife! I shall be a faithful
husband, and when I tell you about my belief and religion, you will
learn that we have the same laws from God, except that we
Christians cannot marry more than one wife. Does such a law
displease you, my Rahma?’
‘Swear,’ she said, ‘that you will never divorce me, never marry
another woman.’
‘I swear,’ he replied, ‘that nought but death shall part us.’
Rahma then threw herself into João’s arms, exclaiming, ‘I am for
ever your loving wife, and shall honour and obey you!’
João and Rahma were very happy. Of an evening, when his work
was done, he taught her to read and write Portuguese, and found
her quick and intelligent in learning. He explained to her the precepts
of the Christian religion, and told her that he hoped the day might
come when he could find some excuse to leave the Moorish Court
and escape with her to Portugal.
When their first child, a girl, was born, Rahma expressed the wish
that her name should be ‘Miriam,’ or Mary, the name of the Mother of
the Saviour of all men, and that she should be brought up in the
Christian faith.
João was very industrious, and continued in high favour with the
Sultan, manufacturing many gun-barrels, upon which, besides his
own name in European characters, he engraved the Arabic word
‘Sidi’ (my Lord), to denote that they were made for the Sultan, and
such barrels are occasionally to be found at the present day.
The Moorish gunsmiths having lost, since João’s arrival at Court,
the Royal custom, took counsel together how they should contrive to
discover the Christian’s secret of forging the twisted barrels; for João
was careful to allow no Moor, except the Sultan, to enter his forge
when he was at work.
 The Portuguese was of very cleanly habits, and had his workshop
whitewashed every month, for which work Jews are usually
employed throughout Morocco. One of the smiths, disguised as a
Jew, offered himself to João to whitewash the forge. He was
engaged, and returned for the same purpose every month.
The sharp-eyed spy watched the operations, and finally learnt so
much of the process as to enable him to imitate it, and he succeeded
so well that he presented a twisted barrel to the Sultan, which His
Majesty considered to be as good as any of João’s make.
The latter was summoned to the Court and asked how it came to
pass that twisted barrels could be made by native gunsmiths. The
unfortunate João declared he had been betrayed by some spy
watching him when at work.
Other Moorish smiths also acquired the art, and, as good barrels
of twisted iron were sold at low prices in Fas, the Sultan discontinued
employing João, and ceased sending him ‘mona’ from the palace.
João, however, had laid by a considerable sum of money, and he
determined to quit the capital with his wife and try to escape to
Tangier. He therefore petitioned the Sultan to be allowed to take his
wife to visit her father, the Sheikh at Beni M’suar.
This was granted, and João bought animals to carry away such
property as he had not been able to dispose of at Fas, and set out
with Rahma and her child for the village of Tsemsalla in the Beni
M’suar mountains.
After remaining some time with his wife at the Sheikh’s house,
where they received a warm welcome, João informed his father-in-
law that he must return to his work. Leaving early one morning with
his wife and child, he proceeded to Tangier, a distance of about
fifteen miles. On arrival at the Portuguese outposts, he was
challenged by a sentry. The soldier proved to be an old comrade
who had heard that João had assumed the disguise of a Moslem,
and, recognising him, allowed him to enter the town, where he was
conducted before the Portuguese Governor, to relate his adventures
and present his wife and child.
The Governor took great interest in João, who had always borne
an excellent character. Rahma, by her husband’s desire, wore the
European dress, and as a Christian no longer veiled her face. The
Governor was much struck by her beauty and gentle manners, and
on learning from her, for she had acquired the Portuguese language,
that she was already converted to the Christian faith and desired to
be baptized by a priest, together with her little girl, he took her to his
wife and daughters, by whom Rahma was made much of. They were
lodged in the Governor’s house, and the baptism was carried out,
with great ceremony, at the Cathedral[45] of Tangier; the child was
christened Miriam.
After a sojourn of some weeks, João and his family were given a
passage in a Government vessel bound to Lisbon, with letters of
recommendation to the King and Queen, to whom their history was
related. The Royal family patronised João, and took especial interest
in pretty Rahma and her daughter as being converts from the
Mohammedan faith.
Being a clever mechanic, João obtained a lucrative employment,
and lived in ease and comfort with his wife, who bore him a large
family.
Rahma wrote to her father and described how happy she and her
husband were, and that they had escaped to the land of the
Nazarenes, as they had feared the jealous and revengeful feelings of
the smiths at the capital; for João, since the betrayal of his secret,
had no longer been shown favour by the Sultan. However, for fear of
causing sorrow to her father, she did not inform him of her
conversion to the Christian faith.
João sent the old Sheikh a beautiful gun, with his own name and
that of Sheikh Shashon engraved on the barrel in letters of gold.
 CHAPTER XIX.

FOURTH MISSION TO MARÁKESH. 1872.

In 1872 Sir John was made Minister Plenipotentiary. This mark of

confidence on the part of Her Majesty’s Government was the more
acceptable as he had recently been attacked in the English press.
The most important of these attacks appeared in the Spectator,
which however afterwards withdrew its charges unreservedly. Unjust
accusations of this nature affected him only for the moment, when
his quick and passionate spirit would fire up under
misrepresentation, for, as he writes: ‘I was lugged out of my little
corner and set on a pedestal to be pelted with dirt—now replaced by
bouquets. I am getting callous to abuse. “Fais ce que dois, advienne
que pourra.”’
In a letter dated September 27, 1872, to Sir Joseph Hooker, he
says:—

They have made me Minister Plenipotentiary, and I am to go to the Moorish

Court to present my new credentials during the winter. The Sultan is at Marákesh,
or will be there when he has ‘eaten up’ a rebel tribe or two. I do not remain
permanently; in fact, I should decline to do so, though I hope the day will come
when we shall have the British Representative resident at the fountain-head, and
thus alone can we hope that the turbid waters may begin to clear.

On March 25, 1873, Sir John, four ladies, and seven gentlemen
embarked on board H.M.S. Lively for Mazagan, en route for
Marákesh. Mazagan, which was reached the following forenoon, has
a picturesque appearance from the sea; but of itself is an
uninteresting town. The country surrounding it is flat and sandy, with
only a few palm-trees and the cupolas of scattered sanctuaries, or
saint-houses, to relieve the monotony of the scenery.
The entrance to the landing-place was by a passage through a
curious old Portuguese breakwater, repaired some years previously
by the Moorish Government at Sir John’s instigation. On landing
under the customary salute, Sir John was welcomed by the
Governor and authorities, who conducted him to the dwelling
prepared for the Mission,—a house standing on what had been,
during the occupation of Mazagan by the Portuguese in the
seventeenth century, the site of a church. Its steeple, now used as a
belvedere, is still standing.
The Sultan had sent a liberal supply of saddle and baggage
animals, and a few extra tents of handsome Moorish make, lined and
decorated within in different coloured cloths. With these were a body
of a dozen ‘fraijia,’ tent-pitchers, attached to his army. These men
proved most efficient and did their work smartly and thoroughly. They
were all, without exception, Bokhári.
The Mission left Mazagan early on the 28th. The escort consisted
of a Kaid Erha and seven officers, with some thirty troopers. ‘Kaid
Erha,’ it may be explained, means ‘the Commander of a Mill,’ as,
during campaigns in Morocco, a hand-mill for grinding corn is allotted
to every thousand men. Hence the title of Kaid Erha given to every
officer in command of a thousand. Kaid el Mia, or Kaid of a hundred,
is the next grade, corresponding to the centurion of the Romans.
Besides this escort, Sir John had with him his own faithful body-
guard of half a dozen men chosen from amongst the Suanni hunters,
men upon whom he could depend in any emergency.
There was no important departure on the journey to Marákesh
from the routine observed on entering the successive provinces. On
each occasion the ‘Bashador’ was received by the Governor or
Khalífa with an escort varying in number, according to the strength
and importance of the province, from about twenty-five to a hundred
men, who invariably indulged in a prolonged display of ‘lab el barod,’
with the inevitable concomitants of dust, noise, and delay. Each
evening too, on arrival in camp, supplies of food in the form of ‘mona’
were brought and presented with the usual formalities. The Sheikh
offered the ‘mona’ in the name of the Sultan, and Sir John always
made a little speech of thanks to the donors.
The route followed for the next two days lay in a south-west
direction, over an undulating country cultivated with wheat, barley,
beans, and maize; and men were ploughing with oxen, or sometimes
even with a camel and donkey yoked together. A little girl followed
each plough dropping ‘dra,’ or millet-seed, into the furrows. Maize is
one of the chiefs exports, since the prohibition of its exportation was
removed at the instance of Sir John in 1871. The soil was a rich,
dark, sandy loam, thickly studded with limestones: these had, in
some parts, been removed and piled up, forming rubble walls round
the crops. Fig-trees and a few palms, scattered here and there,
scarcely relieved the flatness of the landscape.
On entering the hilly country of Erhamna on April 2, two horsemen
of Dukála, with a couple of falcons, joined the cavalcade. They told
Sir John that they had received orders from the Sultan to show him
some sport; but they expressed their fear that the birds would not
strike the game, as it was the moulting season and they were not in
good feather.
A line of horsemen was formed, and, after riding half an hour, a
‘kairwan’ or stone plover was started. The falcon was thrown up, and
soon stooped but missed her quarry. The plover seemed so
paralysed by the attack that it settled in the grass, and was only
compelled with difficulty by the horsemen to rise. In the second flight
the falcon struck the plover, whose throat was cut, and the hawk was
given a few drops of blood. Another trial was made, but the hawks
seemed dull, and only came back and lighted near their masters.
The falconers therefore were dismissed with a gift and many thanks.
Thus the hopes we had entertained of finding a great bustard and
pursuing it with the falcons was not realised, as none were met with.
But, on the return of the sportsmen to the regular track, Miss A. Hay,
who had remained near Lady Hay’s litter, informed them that she
had seen several of these gigantic birds, which had crossed their
path.
Hunting with falcons is in Morocco a Royal sport, and no subject
of the Sultan, unless he be a member of the Royal family, can hunt
with them, without being especially granted the privilege. A few years
before this, the Sultan sent Sir John a gift of two falcons—and with
them a falconer, capable of catching and training others, to instruct
him in the sport. The novelty proved interesting for a time; but in
comparison with pig-sticking, coursing and shooting, it was found
wanting, and the falcons soon ceased to be more than mere pets at
the Legation.
Sir John, who was a great admirer of these birds, used to relate
the following legend and its curious verification in his own personal
experience.

There is a legend that no one of the name of Hay should kill or

injure a falcon. The tradition is founded on the following tale.
At the battle of Loncarty in 980 the Danish army was certainly
routed by the Scots. Yet, at the commencement of this battle, the
Danes had been victorious and drove the Scots before them, pell
mell, towards a narrow pass. Here three stalwart Highlanders, a
father and his two sons, had taken their stand and rallied their
fugitive countrymen. Then, placing themselves at their head, they led
them in an onslaught on the Danes, whom they routed.
Afterwards, the King of the Scots, Kenneth III, sent for the three
men, and, learning from them that they—who were farmers—had
been occupied in ploughing when they saw the Scots in retreat, and
then joined in the fray, he exclaimed, ‘Henceforward you shall be
called Garadh!’ which in Gaelic signifies bulwark or fence. Later this
name was transformed to De la Haye by members of the family who
emigrated to Normandy and, establishing themselves there, joined
the Conqueror when he came to England. Subsequently it was
modified into Hay.
King Kenneth ennobled Garadh, and offered him a grant of land of
his own selection. Garadh prayed the King to grant him whatever
land his falcon might traverse, till it alighted, if thrown off at Loncarty.
His prayer was granted. The falcon flew from Loncarty and alighted
on the Carse of Gowry—as indeed might have been expected, since
Garadh was wont to hunt with falcons and frequently fed his birds on
that height. This large property was long held by the Hay family, but
the greater part passed into other hands during the last century.
My father, who told me this legend, added a caution against ever
injuring the bird which had brought good fortune to the family, and I
bore it in mind, and never fired a shot at any falcon, until one day I
received a letter from a naturalist in England, requesting me to find
some person who would aid him in making a collection of specimens
of birds of prey, as he knew that these birds migrated northwards in
the month of March—when the wind blows from the east—passing
from Morocco, across the Straits, to the Spanish coast, and selecting
generally for the point of their departure the Marshan—a plateau
within a quarter of a mile of Tangier. From here I have seen
hundreds of birds of prey, eagles, falcons, hawks, kestrels, kites and
buzzards cross the Straits during the month of March, flying against
the east wind.
Being desirous of meeting the wishes of my friend the naturalist, I
selected a spot on the Marshan, where, in a dilapidated battery, were
three or four dismounted guns, presented by King George IV to a
former Sultan. Here, ensconced between two of the guns, I waited
the passage of the birds and shot several kites, buzzards, kestrels
and other hawks; but at first, true to my rule, spared the falcons.
It was in the days of muzzle-loaders with copper caps, and I was
not using a gun of English make.
At last, seeing a fine falcon flying towards me, I said to myself,
‘What folly to believe in such silly old-womanish nonsense as that a
Hay must not injure a falcon—I shall test the truth of the legend by
firing at one.’
The bird came towards me, I fired: the gun burst at the breech,
the right-hand nipple flew out, grazing my forehead near my right
eye, and my wrist was burnt. I threw down the gun, exclaiming,
‘Thank God I was not killed! Henceforward I am a believer.’
The falcon was only slightly wounded; some few feathers fell from
the poor bird, and it continued its flight. Had it been killed, I suppose
I should not have lived to tell this story!

Two days later, the party crossed the Beheira u el Gintsor, a

district which, twenty years ago, was uninhabited and full of gazelles,
great bustard, and other game. But the present Sultan had punished
a rebellious tribe by removing them from a rich land and quartering
them on this barren plateau. It is now full of cattle, and patches of
cultivation were to be seen here and there.
The Arabs of the district brought some greyhounds, for the
purpose of hunting hare; but the attempt at sport proved a failure.
Amongst these dogs were two of the native rough-coated breed,
which much resemble the Scotch deer-hound, or sleugh-hound.
Curiously enough, the Arabic word for greyhound (in Morocco) is
slogi or sloki—plural slak. These particular dogs were poor and
stunted in appearance, but sometimes handsome specimens are
met with. They are supposed to be endowed with great powers of
endurance.
 Next day, on ascending the hill of Jebíla, the city of Marákesh
came into view, with its numerous minarets; amongst which towered
the great mosque of the Kutubía—dwarfing all others by comparison.
Through the pass at the foot of the hills, called Birra Burub—
evidently an ancient Berber name—they entered a forest of palms
and crossed the many-arched bridge over the Tensift river. The camp
was pitched on the banks of the river, which, in the swollen torrent,
was racing past—at least a hundred yards wide—carrying with it,
now and then, palm-trees washed away by the flood.
On the 5th of April the Mission entered Marákesh, passing
through the beautiful forest of palms. Soon after leaving camp, they
were met by a body of a hundred cavalry, accompanied by Kaid Bu
Aiesh, the second Chamberlain. He brought a welcome from the
Sultan—‘and a thousand times welcome.’ He added that the troop
which accompanied him was entirely composed of ‘Kaids,’ or
officers, who were sent as a guard of honour to the British
Representative on his entry.
Entering the city by the Bab Hamár, they proceeded to the
summer palace of the Maimunía, where Sir John was received by
the Governor of the city and other officials, and conducted to a
‘kubba’ or small pavilion at the end of a long avenue in the beautiful
garden, or rather orchard, attached to the dwelling. All kinds of fruit-
trees abounded, intermingled with palms and cypresses, and
intersected by broad avenues of large olive-trees. The fragrance of
the orange and lemon-trees in full flower filled the air. The only
flowers were the large white jasmine and the scented single rose of
‘Sigelmasi’—used in making attar of roses; but both these grew in
profusion.
The ‘kubba,’ the Governor said, had been prepared as Sir John’s
bedroom. It was richly carpeted and encircled by divans. A large and
handsome brass bed stood in a recess, while an ugly deal
washstand, apparently made for the occasion, furnished with utensils
of uncouth form and colour, contrasted unfavourably with the
Moorish fittings. After the authorities had taken leave, the other
apartments were investigated and found to be ample and well
furnished in the Moorish style. The doors and ceilings, which were
decorated with arabesque work, carved and coloured, had evidently
been recently repainted. Facing the entrance to the main dwelling
was a beautiful fountain, set in the wall in a horseshoe arch of tiles
and delicate geometric tracery. In the centre of the courtyard, on to
which the rooms opened, was a large marble basin in which bubbled
another jet of water. The archways of the doors were beautifully
decorated with carved filagree work.
On the morning of the 7th, as pre-arranged, Hadj Mohammed Bu
Aiesh, the chief Usher, announced in person that the Sultan would
be prepared to receive Sir John and the members of the Mission at 9
o’clock. This official was attired in the rich dress of a Moorish
courtier. Several coloured cloth caftans, or long tunics, richly
embroidered at the edges and seams with silk, were covered by
another of white cotton with flowing sleeves, and over these was
draped the creamy woollen ‘haik,’ which marks the civilian, of which
the soft folds hung to the ground. His turban of spotless white was
rolled, fold upon fold, above his brow, forming a disk of marvellous
size round the red fez which peeped above it.
Shortly after this announcement, a procession was formed. A
double line of the irregular soldiers in their picturesque and flowing
dress of all the colours of the rainbow, led the way. They were
followed by Sir John, the chief Usher riding on his left, and two
officers of the Askar, or regulars, walking on either side of his horse.
Then came the gentlemen of the Mission, all in uniform. The gates of
the palace precincts had been closed to prevent the mob crowding
in, and were only opened to admit the cortège. In the great court, or
square, were drawn up between three and four thousand Askar, who
presented arms when the ‘Bashador’ appeared.
The scene as usual was brilliant in its barbaric pomp of led horses
handsomely caparisoned, gaily dressed attendants, many-hued
soldiery, and solemn, white-robed officials. But in curious contrast to
the gaiety of his surroundings, stood prominent an old ‘deruish[46],’
with whom no one interfered. He was dirty, ragged and decrepit,
perhaps deranged, for he gazed around with a strange wild air.
During the Sultan’s ceremonious interview with Sir John the ‘deruish’
stood, with uplifted hands, loudly blessing the ‘Prince of believers.’
Next day some of the idlers of the party visited the town.
Accompanied by an escort of fourteen men and an officer, they
made their way to the ‘Mellah’ or Jewish quarter, a horribly dirty
place. The Hebrews of Marákesh are an ugly and apparently
degraded race. To add to their unsightly appearance, the men wear
blue kerchiefs with white spots, tied over their heads and under their
chins. Two long oily curls hang on either side of their faces. Their
greasy cloaks, blue or black, are similar to those worn by the natives
of Sus, and have a curious lozenge-shaped pattern in red and yellow
woven across the back. Tradition relates that these cloaks were first
woven by Spanish captives in the sixteenth century, who worked the
Spanish colours on the back of the cloaks destined for their own use.
The Jewish women, with the exception of a very few young girls,
were no better looking than the men. But their out-door dress is
graceful and pleasing, as they envelop themselves in a large veil of
soft white cotton of native manufacture, bordered with a broad band
of silk—also white—which is arranged to fall in front. Three centuries
ago this veil, with white or coloured silk borders, was worn by the
Moslem women of Marákesh, who now wrap themselves, when they
go abroad, in the more clumsy and less becoming heavy woollen
‘haik.’
The large escort which, when the party started, had been looked
on as an absurd precaution, proved to be really necessary. Though
the people showed no incivility, the pressure of the dense crowds
that thronged after the strangers would have rendered progress
without an escort well nigh impossible.
A few days later the whole party dined at the house of the Hajib—
Sid Musa. They rode thither through the deserted streets in bright
moonlight, which enabled them to avoid the holes and pitfalls
abounding in this decaying town. Well-dressed dependants waited at
Sid Musa’s door to take their horses, and, following a man with a
lantern, they soon found themselves in a small but beautiful court,
with a fountain playing in a marble basin in the centre. Near this
stood five tea-kettles on little charcoal stoves, and as many
diminutive tables, each bearing a tray covered with a silk kerchief—
suggestive of tea. Sid Musa and a Sheríf called the Bakáli, a
favourite of the Sultan, welcomed them, and led the way into a room
furnished with two gorgeous beds, chairs, sofas, and divans covered
with brocade and satin. Handsome mirrors, draped with embroidered
silken scarves, hung round the walls, which were covered with velvet
arras embroidered in gold. These hangings, which cover the lower
portion of the walls of every respectable Moorish dwelling, and vary
in richness of material according to the wealth of the owner, appear
to be a remnant of their ancient life as nomad Arabs. The hanging
resembles the side of the tent still in use among the Moors. The
design is invariably a succession of horse-shoe arches in different
coloured materials and more or less richly embroidered. In mosques
and holy places, and in them alone, mats, often very fine, are used
for the same purpose.
After the guests had been introduced to their hosts and the usual
compliments had passed, in the course of conversation Sir John
expressed to Sid Musa his desire to visit the Atlas Mountains. With
the view of preventing the objections which are often raised by the
Moorish Government when Europeans wish to penetrate into the
more remote regions of Morocco, he observed that he was born and
bred a highlander and that he longed to be once more among
mountains. Sid Musa and the Bakáli, being both mountaineers, quite
concurred in this sentiment and promised to aid in promoting an
expedition.
Dinner was long delayed, and Sid Musa became restless till the
Sheríf informed him that the guest’s servants had been consulted
regarding the feast, and that they had advised the Moorish chef (a
coal-black slave) to reverse the usual order of a native meal; as it
had been intended that the sweet dishes should be served first and
the viands afterwards.