Network Security Through Data

Analysis From Data to Action 2nd

Edition Michael Collins
Visit to download the full and correct content document:
https://textbookfull.com/product/network-security-through-data-analysis-from-data-to-
action-2nd-edition-michael-collins/
More products digital (pdf, epub, mobi) instant
download maybe you interests ...

Optimizing Data-to-Learning-to-Action Steven Flinn

https://textbookfull.com/product/optimizing-data-to-learning-to-
action-steven-flinn/

Categorical and Nonparametric Data Analysis E. Michael

Nussbaum

https://textbookfull.com/product/categorical-and-nonparametric-
data-analysis-e-michael-nussbaum/

R in Action Data Analysis and Graphics with R Bonus ch

23 ONLY 2nd Edition Robert Kabacoff

https://textbookfull.com/product/r-in-action-data-analysis-and-
graphics-with-r-bonus-ch-23-only-2nd-edition-robert-kabacoff/

Data Analysis with R 2nd Edition Tony Fischetti

https://textbookfull.com/product/data-analysis-with-r-2nd-
edition-tony-fischetti/
Probability and Computing Randomization and
Probabilistic Techniques in Algorithms and Data
Analysis 2nd Edition Michael Mitzenmacher

https://textbookfull.com/product/probability-and-computing-
randomization-and-probabilistic-techniques-in-algorithms-and-
data-analysis-2nd-edition-michael-mitzenmacher/

Data Analysis from Scratch with Python Peters Morgan

https://textbookfull.com/product/data-analysis-from-scratch-with-
python-peters-morgan/

From Curve Fitting to Machine Learning An Illustrative

Guide to Scientific Data Analysis and Computational
Intelligence 2nd Edition Achim Zielesny

https://textbookfull.com/product/from-curve-fitting-to-machine-
learning-an-illustrative-guide-to-scientific-data-analysis-and-
computational-intelligence-2nd-edition-achim-zielesny/

Introduction to Quantitative Data Analysis in the

Behavioral and Social Sciences Michael J. Albers

https://textbookfull.com/product/introduction-to-quantitative-
data-analysis-in-the-behavioral-and-social-sciences-michael-j-
albers/

Environmental Data Analysis with Matlab 2nd Edition

William Menke

https://textbookfull.com/product/environmental-data-analysis-
with-matlab-2nd-edition-william-menke/
Praise for Network Security Through Data Analysis, Second
Edition
Attackers generally know our technology better than we do, yet a
defender’s first reflex is usually to add more complexity, which just
makes the understanding gap even wider — we won’t win many
battles that way. Observation is the cornerstone of knowledge, so
we must instrument and characterize our infrastructure if we hope
to detect anomalies and predict attacks. This book shows how and
explains why to observe that which we defend, and ought to be
required reading for all SecOps teams.
Dr. Paul Vixie, CEO of Farsight Security
Michael Collins provides a comprehensive blueprint for where to
look, what to look for, and how to process a diverse array of data
to help defend your organization and detect/deter attackers. It is a
“must have” for any data-driven cybersecurity program.
Bob Rudis, Chief Data Scientist, Rapid7
Combining practical experience, scientific discipline, and a solid
understanding of both the technical and policy implications of
security, this book is essential reading for all network operators
and analysts. Anyone who needs to influence and support decision
making, both for security operations and at a policy level, should
read this.
Yurie Ito, Founder and Executive Director, CyberGreen Institute
Michael Collins brings together years of operational expertise and
research experience to help network administrators and security
analysts extract actionable signals amidst the noise in network
logs. Collins does a great job of combining the theory of data
analysis and the practice of applying it in security contexts using
real-world scenarios and code.
Vyas Sekar, Associate Professor, Carnegie Mellon University/CyLab
Network Security Through
Data Analysis
From Data to Action

Michael Collins
Network Security Through Data Analysis
by Michael Collins
Copyright © 2017 Michael Collins. All rights reserved.
Printed in the United States of America.
Published by O’Reilly Media, Inc., 1005 Gravenstein Highway North,
Sebastopol, CA 9547.
O’Reilly books may be purchased for educational, business, or sales
promotional use. Online editions are also available for most titles
(http://oreilly.com/safari). For more information, contact our
corporate/institutional sales department: 800-998-9938 or
corporate@oreilly.com.
Editors: Courtney Allen and Virginia Wilson

Production Editor: Nicholas Adams

Copyeditor: Rachel Head

Proofreader: Kim Cofer

Indexer: WordCo Indexing Services, Inc.

Interior Designer: David Futato

Cover Designer: Karen Montgomery

Illustrator: Rebecca Demarest

February 2014: First Edition

September 2017: Second Edition

Revision History for the Second Edition
2017-09-08: First Release

See http://oreilly.com/catalog/errata.csp?isbn=9781491962848 for

release details.
The O’Reilly logo is a registered trademark of O’Reilly Media, Inc.
Network Security Through Data Analysis, the cover image, and
related trade dress are trademarks of O’Reilly Media, Inc.
While the publisher and the author have used good faith efforts to
ensure that the information and instructions contained in this work
are accurate, the publisher and the author disclaim all responsibility
for errors or omissions, including without limitation responsibility for
damages resulting from the use of or reliance on this work. Use of
the information and instructions contained in this work is at your
own risk. If any code samples or other technology this work contains
or describes is subject to open source licenses or the intellectual
property rights of others, it is your responsibility to ensure that your
use thereof complies with such licenses and/or rights.
978-1-491-96284-8
[LSI]
Preface

This book is about networks: monitoring them, studying them, and

using the results of those studies to improve them. “Improve” in this
context hopefully means to make more secure, but I don’t believe
we have the vocabulary or knowledge to say that confidently — at
least not yet. In order to implement security, we must know what
decisions we can make to do so, which ones are most effective to
apply, and the impact that those decisions will have on our users.
Underpinning these decisions is a need for situational awareness.
Situational awareness, a term largely used in military circles, is
exactly what it says on the tin: an understanding of the environment
you’re operating in. For our purposes, situational awareness
encompasses understanding the components that make up your
network and how those components are used. This awareness is
often radically different from how the network is configured and how
the network was originally designed.
To understand the importance of situational awareness in
information security, I want you to think about your home, and I
want you to count the number of web servers in your house. Did you
include your wireless router? Your cable modem? Your printer? Did
you consider the web interface to CUPS? How about your television
set?
To many IT managers, several of the devices just listed won’t have
registered as “web servers.” However, most modern embedded
devices have dropped specialized control protocols in favor of a web
interface — to an outside observer, they’re just web servers, with
known web server vulnerabilities. Attackers will often hit embedded
systems without realizing what they are — the SCADA system is a
Windows server with a couple of funny additional directories, and
the MRI machine is a perfectly serviceable spambot.
This was all an issue when I wrote the first edition of the book; at
the time, we discussed the risks of unpatched smart televisions and
vulnerabilities in teleconferencing systems. Since that time, the
Internet of Things (IoT) has become even more of a thing, with
millions of remotely accessible embedded devices using simple (and
insecure) web interfaces.
This book is about collecting data and looking at networks in order
to understand how the network is used. The focus is on analysis,
which is the process of taking security data and using it to make
actionable decisions. I emphasize the word actionable here because
effectively, security decisions are restrictions on behavior. Security
policy involves telling people what they shouldn’t do (or, more
onerously, telling people what they must do). Don’t use a public file
sharing service to hold company data, don’t use 123456 as the
password, and don’t copy the entire project server and sell it to the
competition. When we make security decisions, we interfere with
how people work, and we’d better have good, solid reasons for
doing so.
All security systems ultimately depend on users recognizing and
accepting the tradeoffs — inconvenience in exchange for safety —
but there are limits to both. Security rests on people: it rests on the
individual users of a system obeying the rules, and it rests on
analysts and monitors identifying when rules are broken. Security is
only marginally a technical problem — information security involves
endlessly creative people figuring out new ways to abuse technology,
and against this constantly changing threat profile, you need
cooperation from both your defenders and your users. Bad security
policy will result in users increasingly evading detection in order to
get their jobs done or just to blow off steam, and that adds
additional work for your defenders.
The emphasis on actionability and the goal of achieving security is
what differentiates this book from a more general text on data
science. The section on analysis proper covers statistical and data
analysis techniques borrowed from multiple other disciplines, but the
overall focus is on understanding the structure of a network and the
decisions that can be made to protect it. To that end, I have
abridged the theory as much as possible, and have also focused on
mechanisms for identifying abusive behavior. Security analysis has
the unique problem that the targets of observation are not only
aware they’re being watched, but are actively interested in stopping
it if at all possible.

THE MRI AND THE GENERAL’S LAPTOP

Several years ago, I talked with an analyst who focused primarily on a university
hospital. He informed me that the most commonly occupied machine on his network
was the MRI. In retrospect, this is easy to understand.
“Think about it,” he told me. “It’s medical hardware, which means it’s certified to use a
specific version of Windows. So every week, somebody hits it with an exploit, roots it,
and installs a bot on it. Spam usually starts around Wednesday.” When I asked why he
didn’t just block the machine from the internet, he shrugged and told me the doctors
wanted their scans. He was the first analyst I’d encountered with this problem, but he
wasn’t the last.
We see this problem a lot in any organization with strong hierarchical figures: doctors,
senior partners, generals. You can build as many protections as you want, but if the
general wants to borrow the laptop over the weekend and let his granddaughter play
Neopets, you’ve got an infected laptop to fix on Monday.

I am a firm believer that the most effective way to defend networks

is to secure and defend only what you need to secure and defend. I
believe this is the case because information security will always
require people to be involved in monitoring and investigation — the
attacks change too frequently, and when we automate defenses,
attackers figure out how to use them against us.1
I am convinced that security should be inconvenient, well defined,
and constrained. Security should be an artificial behavior extended
to assets that must be protected. It should be an artificial behavior
because the final line of defense in any secure system is the people
in the system — and people who are fully engaged in security will be
mistrustful, paranoid, and looking for suspicious behavior. This is not
a happy way to live, so in order to make life bearable, we have to
limit security to what must be protected. By trying to watch
everything, you lose the edge that helps you protect what’s really
important.
Because security is inconvenient, effective security analysts must be
able to convince people that they need to change their normal
operations, jump through hoops, and otherwise constrain their
mission in order to prevent an abstract future attack from
happening. To that end, the analysts must be able to identify the
decision, produce information to back it up, and demonstrate the
risk to their audience.
The process of data analysis, as described in this book, is focused on
developing security knowledge in order to make effective security
decisions. These decisions can be forensic: reconstructing events
after the fact in order to determine why an attack happened, how it
succeeded, or what damage was done. These decisions can also be
proactive: developing rate limiters, intrusion detection systems
(IDSs), or policies that can limit the impact of an attacker on a
network.
Audience
The target audience for this book is network administrators and
operational security analysts, the personnel who work on NOC floors
or who face an IDS console on a regular basis. Information security
analysis is a young discipline, and there really is no well-defined
body of knowledge I can point to and say, “Know this.” This book is
intended to provide a snapshot of analytic techniques that I or other
people have thrown at the wall over the past 10 years and seen
stick. My expectation is that you have some familiarity with TCP/IP
tools such as netstat, tcpdump, and wireshark.
In addition, I expect that you have some familiarity with scripting
languages. In this book, I use Python as my go-to language for
combining tools. The Python code is illustrative and might be
understandable without a Python background, but it is assumed that
you possess the skills to create filters or other tools in the language
of your choice.
In the course of writing this book, I have incorporated techniques
from a number of different disciplines. Where possible, I’ve included
references back to original sources so that you can look through that
material and find other approaches. Many of these techniques
involve mathematical or statistical reasoning that I have intentionally
kept at a functional level rather than going through the derivations
of the approach. A basic understanding of statistics will, however, be
helpful.
Contents of This Book
This book is divided into three sections: Data, Tools, and Analytics.
The Data section discusses the process of collecting and organizing
data. The Tools section discusses a number of different tools to
support analytical processes. The Analytics section discusses
different analytic scenarios and techniques. Here’s a bit more detail
on what you’ll find in each.
Part I discusses the collection, storage, and organization of data.
Data storage and logistics are critical problems in security analysis;
it’s easy to collect data, but hard to search through it and find actual
phenomena. Data has a footprint, and it’s possible to collect so much
data that you can never meaningfully search through it. This section
is divided into the following chapters:
Chapter 1
This chapter discusses the general process of collecting data. It
provides a framework for exploring how different sensors collect
and report information and how they interact with each other,
and how the process of data collection affects the data collected
and the inferences made.
Chapter 2
This chapter expands on the discussion in the previous chapter
by focusing on sensor placement in networks. This includes
points about how packets are transferred around a network and
the impact on collecting these packets, and how various types
of common network hardware affect data collection.
Chapter 3
This chapter focuses on the data collected by network sensors
including tcpdump and NetFlow. This data provides a
comprehensive view of network activity, but is often hard to
interpret because of difficulties in reconstructing network traffic.
Chapter 4
This chapter focuses on the process of data collection in the
service domain — the location of service log data, expected
formats, and unique challenges in processing and managing
service data.
Chapter 5
This chapter focuses on the data collected by service sensors
and provides examples of logfile formats for major services,
particularly HTTP.
Chapter 6
This chapter discusses host-based data such as memory and
disk information. Given the operating system–specific
requirements of host data, this is a high-level overview.
Chapter 7
This chapter discusses data in the active domain, covering
topics such as scanning hosts and creating web crawlers and
other tools to probe a network’s assets to find more information.
Part II discusses a number of different tools to use for analysis,
visualization, and reporting. The tools described in this section are
referenced extensively in the third section of the book when
discussing how to conduct different analytics. There are three
chapters on tools:

Chapter 8
This chapter is a high-level discussion of how to collect and
analyze security data, and the type of infrastructure that should
be put in place between sensor and SIM.
Chapter 9
 The System for Internet-Level Knowledge (SiLK) is a flow
analysis toolkit developed by Carnegie Mellon’s CERT Division.
This chapter discusses SiLK and how to use the tools to analyze
NetFlow, IPFIX, and similar data.
Chapter 10
One of the more common and frustrating tasks in analysis is
figuring out where an IP address comes from. This chapter
focuses on tools and investigation methods that can be used to
identify the ownership and provenance of addresses, names,
and other tags from network traffic.
Part III introduces analysis proper, covering how to apply the tools
discussed throughout the rest of the book to address various
security tasks. The majority of this section is composed of chapters
on various constructs (graphs, distance metrics) and security
problems (DDoS, fumbling):

Chapter 11
Exploratory data analysis (EDA) is the process of examining data
in order to identify structure or unusual phenomena. Both
attacks and networks are moving targets, so EDA is a necessary
skill for any analyst. This chapter provides a grounding in the
basic visualization and mathematical techniques used to explore
data.
Chapter 12
Log data, payload data — all of it is likely to include some forms
of text. This chapter focuses on the encoding and analysis of
semistructured text data.
Chapter 13
This chapter looks at mistakes in communications and how
those mistakes can be used to identify phenomena such as
scanning.
Chapter 14
 This chapter discusses analyses that can be done by examining
traffic volume and traffic behavior over time. This includes
attacks such as DDoS and database raids, as well as the impact
of the workday on traffic volumes and mechanisms to filter
traffic volumes to produce more effective analyses.
Chapter 15
This chapter discusses the conversion of network traffic into
graph data and the use of graphs to identify significant
structures in networks. Graph attributes such as centrality can
be used to identify significant hosts or aberrant behavior.
Chapter 16
This chapter discusses the unique problems involving insider
threat data analysis. For network security personnel, insider
threat investigations often require collecting and comparing data
from a diverse and usually poorly maintained set of data
sources. Understanding what to find and what’s relevant is
critical to handling this trying process.
Chapter 17
Threat intelligence supports analysis by providing
complementary and contextual information to alert data.
However, there is a plethora of threat intelligence available, of
varying quality. This chapter discusses how to acquire threat
intelligence, vet it, and incorporate it into operational analysis.
Chapter 19
This chapter discusses a step-by-step process for inventorying a
network and identifying significant hosts within that network.
Network mapping and inventory are critical steps in information
security and should be done on a regular basis.
Chapter 20
Operational security is stressful and time-consuming; this
chapter discusses how analysis teams can interact with
operational teams to develop useful defenses and analysis
techniques.
Changes Between Editions
The second edition of this book takes cues from the feedback I’ve
received from the first edition and the changes that have occurred in
security since the time I wrote it. For readers of the first edition, I
expect you’ll find about a third of the material is new. These are the
most significant changes:
I have removed R from the examples, and am now using Python
(and the Anaconda stack) exclusively. Since the previous edition,
Python has acquired significant and mature data analysis tools.
This also saves space on language tutorials which can be spent
on analytics discussions.

The discussions of host and active domain data have been

expanded, with a specific focus on the information that a
network security analyst needs. Much of the previous IDS
material has been moved into those chapters.

I have added new chapters on several topics, including text

analysis, insider threat, and interacting with operational
communities.

Most of the new material is based around the idea of an analysis

team that interacts with and supports the operations team. Ideally,
the analysis team has some degree of separation from operational
workflow in order to focus on longer-term and larger issues such as
tools support, data management, and optimization.
 TOOLS OF THE TRADE
So, given Python, R, and Excel, what should you learn? If you expect to focus purely
on statistical and numerical analysis, or you work heavily with statisticians, learn R
first. If you expect to integrate tightly with external data sources, use techniques that
aren’t available in CRAN, or expect to do something like direct packet manipulation or
server integration, learn Python (ideally iPython and Pandas) first. Then learn Excel,
whether you want to or not. Once you’ve learned Excel, take a nice vacation and then
learn whatever tool is left of these three.
All of these data analysis environments provide common tools: some equivalent of a
data frame, visualization, and statistical functionality. Of the three, the Pandas stack
(that is, Python, NumPy, SciPy, Matplotlib, and supplements) provides the greatest
variety of tools, and if you’re looking for something outside of the statistical domain,
Python is going to have it. R, in comparison, is a tightly integrated statistical package
where you will always find the latest statistical analysis and machine learning tools.
The Pandas stack involves combining multiple toolsets developed in parallel, resulting
in both redundancy and valuable tools located all over the place. R, on the other hand,
inherits from this parallel development community (via S and SAS) and sits in the
developer equivalent of the Uncanny Valley.
So why Excel? Because operational analysts live and die off of Excel spreadsheets.
Excel integration (even if it’s just creating a button to download a CSV of your results)
will make your work relevant to the operational floor. Maybe you do all your work in
Python, but at the end, if you want analysts to use it, give them something they can
plunk into a spreadsheet.
Conventions Used in This Book
The following typographical conventions are used in this book:

Italic
Indicates new terms, URLs, email addresses, filenames, and file
extensions.
Constant width
Used for program listings, as well as within paragraphs to refer
to program elements such as variable or function names,
databases, data types, environment variables, statements, and
keywords. Also used for commands and command-line utilities,
switches, and options.
Constant width bold
Shows commands or other text that should be typed literally by
the user.
Constant width italic
Shows text that should be replaced with user-supplied values or
by values determined by context.
Using Code Examples
Supplemental material (code examples, exercises, etc.) is available
for download at https://github.com/mpcollins/nsda_examples.
This book is here to help you get your job done. In general, if
example code is offered with this book, you may use it in your
programs and documentation. You do not need to contact us for
permission unless you’re reproducing a significant portion of the
code. For example, writing a program that uses several chunks of
code from this book does not require permission. Selling or
distributing a CD-ROM of examples from O’Reilly books does require
permission. Answering a question by citing this book and quoting
example code does not require permission. Incorporating a
significant amount of example code from this book into your
product’s documentation does require permission.
We appreciate, but do not require, attribution. An attribution usually
includes the title, author, publisher, and ISBN. For example:
“Network Security Through Data Analysis by Michael Collins
(O’Reilly). Copyright 2017 Michael Collins, 978-1-491-96284-8.”
If you feel your use of code examples falls outside fair use or the
permission given above, feel free to contact us at
permissions@oreilly.com.
O’Reilly Safari
NOTE
Safari (formerly Safari Books Online) is a membership-based training
and reference platform for enterprise, government, educators, and
individuals.
Members have access to thousands of books, training videos,
Learning Paths, interactive tutorials, and curated playlists from over
250 publishers, including O’Reilly Media, Harvard Business Review,
Prentice Hall Professional, Addison-Wesley Professional, Microsoft
Press, Sams, Que, Peachpit Press, Adobe, Focal Press, Cisco Press,
John Wiley & Sons, Syngress, Morgan Kaufmann, IBM Redbooks,
Packt, Adobe Press, FT Press, Apress, Manning, New Riders,
McGraw-Hill, Jones & Bartlett, and Course Technology, among
others.
For more information, please visit http://oreilly.com/safari.
How to Contact Us
Please address comments and questions concerning this book to the
publisher:
O’Reilly Media, Inc.

1005 Gravenstein Highway North

Sebastopol, CA 95472

800-998-9938 (in the United States or Canada)

707-829-0515 (international or local)

707-829-0104 (fax)

We have a web page for this book, where we list errata, examples,
and any additional information. You can access this page at
http://bit.ly/nstda2e.
To comment or ask technical questions about this book, send email
to bookquestions@oreilly.com.
For more information about our books, courses, conferences, and
news, see our website at http://www.oreilly.com.
Find us on Facebook: http://facebook.com/oreilly
Follow us on Twitter: http://twitter.com/oreillymedia
Watch us on YouTube: http://www.youtube.com/oreillymedia
Acknowledgments
I need to thank my editors, Courtney Allen, Virginia Wilson, and
Maureen Spencer, for their incredible support and feedback, without
which I would still be rewriting commentary on regression over and
over again. I also want to thank my assistant editors, Allyson
MacDonald and Maria Gulick, for riding herd and making me get the
thing finished. I also need to thank my technical reviewers: Markus
DeShon, André DiMino, and Eugene Libster. Their comments helped
me to rip out more fluff and focus on the important issues.
This book is an attempt to distill down a lot of experience on ops
floors and in research labs, and I owe a debt to many people on
both sides of the world. In no particular order, this includes Jeff
Janies, Jeff Wiley, Brian Satira, Tom Longstaff, Jay Kadane, Mike
Reiter, John McHugh, Carrie Gates, Tim Shimeall, Markus DeShon,
Jim Downey, Will Franklin, Sandy Parris, Sean McAllister, Greg Virgin,
Vyas Sekar, Scott Coull, and Mike Witt.
Finally, I want to thank my mother, Catherine Collins.

1 Consider automatically locking out accounts after x number of failed password attempts,
and combine it with logins based on email addresses. Consider how many accounts an
attacker can lock out that way.
 Part I. Data

This section discusses the collection and storage of data for use in
analysis and response. Effective security analysis requires collecting
data from widely disparate sources, each of which provides part of a
picture about a particular event taking place on a network.
To understand the need for hybrid data sources, consider that most
modern bots are general-purpose software systems. A single bot
may use multiple techniques to infiltrate and attack other hosts on a
network. These attacks may include buffer overflows, spreading
across network shares, and simple password cracking. A bot
attacking an SSH server with a password attempt may be logged by
that host’s SSH logfile, providing concrete evidence of an attack but
no information on anything else the bot did. Network traffic might
not be able to reconstruct the sessions, but it can tell you about
other actions by the attacker — including, say, a successful long
session with a host that never reported such a session taking place,
no siree.
The core challenge in data-driven analysis is to collect sufficient data
to reconstruct rare events without collecting so much data as to
make queries impractical. Data collection is surprisingly easy, but
making sense of what’s been collected is much harder. In security,
this problem is complicated by the rare actual security threats.
Attacks are common, threats are rare. The majority of network
traffic is innocuous and highly repetitive: mass emails, everyone
watching the same YouTube video, file accesses. Interspersed
among this traffic are attacks, but the majority of the attacks will be
automated and unsubtle: scanning, spamming, and the like. Within
those attacks will be a minority, a tiny subset representing actual
threats.
That security is driven by rare, small threats means that almost all
security analysis is I/O bound: to find phenomena, you have to
search data, and the more data you collect, the more you have to
search. To put some concrete numbers on this, consider an OC-3: a
single OC-3 can generate 5 terabytes of raw data per day. By
comparison, an eSATA interface can read about 0.3 gigabytes per
second, requiring several hours to perform one search across that
data, assuming that you’re reading and writing data across different
disks. The need to collect data from multiple sources introduces
redundancy, which costs additional disk space and increases query
times. It is completely possible to instrument oneself blind.
A well-designed storage and query system enables analysts to
conduct arbitrary queries on data and expect a response within a
reasonable time frame. A poorly designed one takes longer to
execute the query than it took to collect the data. Developing a good
design requires understanding how different sensors collect data;
how they complement, duplicate, and interfere with each other; and
how to effectively store this data to empower analysis. This section
is focused on these problems.
This section is divided into seven chapters. Chapter 1 is an
introduction to the general process of sensing and data collection,
and introduces vocabulary to describe how different sensors interact
with each other. Chapter 2 discusses the collection of network data
— its value, points of collection, and the impact of vantage on
network data collection. Chapter 3 discusses sensors and outputs.
Chapter 4 focuses on service data collection and vantage. Chapter 5
focuses on the content of service data — logfile data, its format, and
converting it into useful forms. Chapter 6 is concerned with host-
based data, such as memory or filesystem state, and how that
affects network data analysis. Chapter 7 discusses active domain
data, scanning and probing to find out what a host is actually doing.
Chapter 1. Organizing Data:
Vantage, Domain, Action, and
Validity

Security analysis is the process of applying data to make security

decisions. Security decisions are disruptive and restrictive —
disruptive because you’re fixing something, restrictive because
you’re constraining behavior. Effective security analysis requires
making the right decision and convincing a skeptical audience that
this is the right decision. The foundations of these decisions are
quality data and quality reasoning; in this chapter, I address both.
Security monitoring on a modern network requires working with
multiple sensors that generate different kinds of data and are
created by many different people for many different purposes. A
sensor can be anything from a network tap to a firewall log; it is
something that collects information about your network and can be
used to make judgment calls about your network’s security.
I want to pull out and emphasize a very important point here:
quality source data is integral to good security analysis.
Furthermore, the effort spent acquiring a consistent source of quality
data will pay off further down the analysis pipeline — you can use
simpler (and faster) algorithms to identify phenomena, you’ll have
an easier time verifying results, and you’ll spend less time cross-
correlating and double-checking information.
So, now that you’re raring to go get some quality data, the question
obviously pops up: what is quality data? The answer is that security
data collection is a trade-off between expressiveness and speed —
packet capture (pcap) data collected from a span port can tell you if
someone is scanning your network, but it’s going to also produce
terabytes of unreadable traffic from the HTTPS server you’re
watching. Logs from the HTTPS server will tell you about file
accesses, but nothing about the FTP interactions going on as well.
The questions you ask will also be situational — how you decide to
deal with an advanced persistent threat (APT) is a function of how
much risk you face, and how much risk you face will change over
time.
That said, there are some basic goals we can establish about
security data. We would like the data to express as much
information with as small a footprint as possible — so data should be
in a compact format, and if different sensors report the same event,
we would like those descriptions to not be redundant. We want the
data to be as accurate as possible as to the time of observation, so
information that is transient (such as the relationships between IP
addresses and domain names) should be recorded at the time of
collection. We also would like the data to be expressive; that is, we
would like to reduce the amount of time and effort an analyst needs
to spend cross-referencing information. Finally, we would like any
inferences or decisions in the data to be accountable; for example, if
an alert is raised because of a rule, we want to know the rule’s
history and provenance.
While we can’t optimize for all of these criteria, we can use them as
guidance for balancing these requirements. Effective monitoring will
require juggling multiple sensors of different types, which treat data
differently. To aid with this, I classify sensors along three attributes:

Vantage
The placement of sensors within a network. Sensors with
different vantages will see different parts of the same event.
Domain
 The information the sensor provides, whether that’s at the host,
a service on the host, or the network. Sensors with the same
vantage but different domains provide complementary data
about the same event. For some events, you might only get
information from one domain. For example, host monitoring is
the only way to find out if a host has been physically accessed.
Action
How the sensor decides to report information. It may just record
the data, provide events, or manipulate the traffic that produces
the data. Sensors with different actions can potentially interfere
with each other.
This categorization serves two purposes. First, it provides a way to
break down and classify sensors by how they deal with data. Domain
is a broad characterization of where and how the data is collected.
Vantage informs us of how the sensor placement affects collection.
Action details how the sensor actually fiddles with data. Together,
these attributes provide a way to define the challenges data
collection poses to the validity of an analyst’s conclusions.
Validity is an idea from experimental design, and refers to the
strength of an argument. A valid argument is one where the
conclusion follows logically from the premise; weak arguments can
be challenged on multiple axes, and experimental design focuses on
identifying those challenges. The reason security people should care
about it goes back to my point in the introduction: security analysis
is about convincing an unwilling audience to reasonably evaluate a
security decision and choose whether or not to make it.
Understanding the validity and challenges to it produces better
results and more realistic analyses.
Domain
We will now examine domain, vantage, and action in more detail. A
sensor’s domain refers to the type of data that the sensor generates
and reports. Because sensors include antivirus (AV) and similar
systems, where the line of reasoning leading to a message may be
opaque, the analyst needs to be aware that these tools import their
own biases.
Table 1-1 breaks down the four major domain classes used in this
book. This table divides domains by the event model and the sensor
uses, with further description following.

Table 1-1. The four domain classes

Domain Data sources Timing Identity

Network PCAP, NetFlow Real-time, packet-based IP, MAC

Service Logs Real-time, event-based IP, Service-based IDs

Host System state, signature alerts Asynchronous IP, MAC, UUID

Active Scanning User-driven IP, Service-based IDs

Sensors operating in the network domain derive all of their data

from some form of packet capture. This may be straight pcap,
packet headers, or constructs such as NetFlow. Network data gives
the broadest view of a network, but it also has the smallest amount
of useful data relative to the volume of data collected. Network
domain data must be interpreted, it must be readable,1 and it must
be meaningful; network traffic contains a lot of garbage.
Sensors in the service domain derive their data from services.
Examples of services include server applications like nginx or apache
(HTTP daemons), as well as internal processes like syslog and the
processes that are moderated by it. Service data provides you with
information on what actually happened, but this is done by
interpreting data and providing an event model that may be only
tangentially related to reality. In addition, to collect service data, you
need to know the service exists, which can be surprisingly difficult to
find out, given the tendency for hardware manufacturers to shop
web servers into every open port.
Sensors in the host domain collect information on the host’s state.
For our purposes, these types of tools fit into two categories:
systems that provide information on system state such as disk
space, and host-based intrusion detection systems such as file
integrity monitoring or antivirus systems. These sensors will provide
information on the impact of actions on the host, but are also prone
to timing issues — many of the state-based systems provide alerts at
fixed intervals, and the intrusion-based systems often use huge
signature libraries that get updated sporadically.
Finally, the active domain consists of sensing controlled by the
analyst. This includes scanning for vulnerabilities, mapping tools
such as traceroute, or even something as simple as opening a
connection to a new web server to find out what the heck it does.
Active data also includes beaconing and other information that is
sent out to ensure that we know something is happening.
Another random document with
no related content on Scribd:
 In the Evening from six to nine, we saw those Appearances in the
Sky called Capræ saltantes, by the Sailors Morrice-Dancers; they
are Streams of Light that suddenly shoot into one another, and
disappear for a Minute or two; yet shifting their Stations within the
Quarter, in so quick and surprizing a manner as might easily deceive
superstitious Times into a belief of Armies in the Air; these, the
Scintillæ volantes, and such like nitrous Exhalations, having given
rise, it’s probable, to all those Prodigies the Air has heretofore in
impious Times abounded with.
The Western Extremity of England that we are now passing by,
has been supposed, from the equal Depth of Water found there,
from Doors, Windows, and Roots of Trees, formerly (it’s said)
hooked up by Fishermen, to have been in Ages past continuous with
the rocky little Islands of Scilly, by a Land called Lioness. When I
consider the Changes Earthquakes and Inundations have made, and
continue insensibly to make on all the different Coasts of the Earth,
losing in some places, and gaining in others; and what new Islands
have now and then been thrust up on the surface of the Waters by
Streams and Currents, subterranean Winds and Fires; the thing
does not appear to me altogether conjectural: the Rocks seem now
with terrible accent to lament the reparation. Who knows but we
likewise are severing eternally from our Friends! it is a Voyage we
shall at some time or other make; and those solitary Rocks that
bound the last sight to our Homes and Countries, naturally bring to
my mind some Reflections on the subject.
Whether when we have shut up this Life we shall remain resolved
into our Elements; revive again in some Plant or Animal; or thirdly,
be reinstated Soul and Body into Glory, is an Enquiry worthy our
utmost Concern and Diligence; as it will sweeten the imbittered
Potions of Life, make us patient under Afflictions, and even easy on
this treacherous Element the Sea; whom none ought to trust; but
they who have a Faith in Providence.
Immortality has been the Ambition of the greatest and wisest Men
that have lived, and indeed who would not in the satiety of worldly
Objects seek out and desire such a prerogative to his Soul? The
Philosophers, rather than lose so comfortable a prospect, have
placed it in an Existence that can never concern our Happiness or
Misery as Men: The Princes of them having made it to subsist in the
universal Soul of the World; from whence, say they, are struck out
Scintillations to every thing that has Life; and in Death, what was
Elementary, returned to its proper station, and what was Divine in us,
to what is Divine in the Universe: a Metempsychosis or
Transmigration continuing the World on in the order we see it.
When I ponder on the natural Cause of our Being ... On the
Necessity of that Cause to produce us, and what the material Agent
is ... On our gradual Advances and Decay; both in respect to Body
and Soul ... The Soul’s sympathizing with the disorders of the Body
... Our Sleeps and total Forgetfulness ... Our Susceptibility of
Madness or Idiocy, and hebridous Productions, especially that of
Man with Beast; I am tempted to think this the most plausible of all
Philosophical Opinions, in relation to our future Existence, that we
are not Creatures of that consequence we imagine; our Natures
neither deserving, nor should they expect in reason any other
Immortality than what other Creatures enjoy in their Seed and
Transmigration. This Doctrine is what the Eastern Sages, reverenced
for their Wisdom, do still propagate, and teach the Heterodox a
Lesson of Humility, That Pride was not made for Man: but at the
same time it makes me superstitiously fear and abhor a Grave of
Waters; which I fancy will subside us too far from the Sun, whose
Power is the chief Principle to revive us again in some Plant or
Animal on our native Element: which will be such perhaps as, in our
Life-time, our Senses were familiar and delighted with; and for which,
we had a sympathetical Affection and Tendency to. But then granting
the Doctrine, I am considering what the Advantage of such a belief
can be to us as Men? Why none. Our personal Identity must be
destroyed in the first Transmigration, much more in an infinite
Succession; and we (as we) can never after this Life be affected with
either Pleasure or Pain. I am therefore on the whole fond of
Revelation, and wish that to be Truth which ascribes such Power to
Faith, That it can remove Mountains: and therefore, with its fruits
concurring, may raise us into immortal Spirits, translate us to Bliss,
even without passing through the horrors of Death.
 That an Almighty Power can subsist us to Eternity, we are very
sure; and that he will do it, I say let it be our Happiness, that we have
the Promises of his Gospel; for here only can be a remedy to all
worldly Cares; and wherever Death o’er takes us, whether this
Voyage or next, so we be upon our Watch, it will transport us to a
Port and Treasure fixed. It will free us from Rocks, Sands, and
tempestuous Seas, and anchor us in a Haven of Felicity.

The Gale with which we left England, carried us the length of Cape
[3]
Finisterre into serener Weather, and Sun-shine; but there we met
with continued Westerly Winds (very unusual to the Coast of
Portugal) which prolonged our Passage. A Day or two’s sail from
Madeira, we fell in with Commodore Matthews, in the Lion, bound
with a Squadron of four Sail to the East-Indies, on the like Service
with ours to Africa, viz. the Suppression of Pyrates.
Abundance of Sea-weed floated about us at 40 Leagues distance,
and continued a constant float till we reached the Island; an
Argument that the bottom of the Sea, especially where the Depths
are decreasing towards any Shore, have a Cloathing of Plants,
which are probably the common Nutriment of large Fish. This our
Divers in Pearl, and Coral-fishing, have confirmed to 8 or 10 Fathom
water; and this, I think, the present Observation proves to be in
greater Depths; 1st, Because the Unwieldiness of some, and the
manner of being provided for Mastication in others, declares
Ruminating, and not Prey, to be the way of Subsistence in many.
2dly, There is a greater Resort toward Shores, than in the distant
Ocean, and perhaps, like many little Fish in our own Channels, they
may have their Seasons of Rotation, and their Grazing, the Cause of
unrooting and throwing it up here. 3dly, Porpoises play about us
daily in Shoals, the most familiar great Fish in the Atlantick, and at all
Parts of it: They tumble most upon a rough Surface, and against the
Wind. Sailors observing these Porpoises, say, they portend Storms.
The Latins call them Porci marini, from some Resemblance to the
Hog, in it’s Entrails and Bigness, (weighing several Hundred.) These
Fish, as they are very numerous, never enticed to the Hook, wasting
many of their hours in play, and gradually lessening from Shore,
shew they know readily where to make their Meals at the bottom of
the Sea, tho’ at other times they certainly prey on smaller and
particular Species of Fish: These their Feasts, it’s like, and That their
ordinary Diet.
 MADEIRA.
This Island, at the first Discovery of it by the Portuguese, about the
Year 1420, was over-run with Wood, whence it’s Name. Divided to
the two Discoverers, they set the Woods on fire, which Travellers say
burnt seven Years; the Ashes giving a vast Fertility to their Sugar-
Canes, at the first Planting; till a Worm getting into the Cane, spoiled
the Increase, &c. so that it is now entirely planted with Vines brought
originally from Candia, which yield the strongest Wines: That called
Malmsey is a rich Cordial, the best made at the Jesuit’s Garden in
Fonchial. Their Vintage is in September and October, and make
about 25000 Pipes.
Others say, one Mecham an Englishman, in a Voyage to Spain
was drove on this Island before the Discovery above: That his Crew
sailed without him and his Mistress; whom he buried here, left an
Inscription on her Tomb, and then in a Canoo of his own building
sailed to Barbary; the King presenting him as a Prodigy to the King
of Castile: From whole Account, the Spaniard soon after made
conquest of the neighbouring Canary Islands. The Island is rocky
Mountains, with an Intermixture of little fruitful Plains. The highest
Parts, Goat-herds and Woods; the Middle, Kitchen-Gardens; and the
Bottom, Vineyards. The Roads bad, which makes them bring their
Wines to town in Hog-skins upon Asses; a brownish and a red sort,
the latter called Vino tinto, being according to common report stained
with Tint, tho’ they assure you it is the natural Grape. They are
almost all limed, a Preservative against the excessive Heats of the
West-Indies, where they are for the most part transported by us, and
where no other Wine keeps well.
Trade is carried on by Bartering, 40 or 50 per Cent. being allowed
on an Invoice of Provisions, Cloaths, or Houshold-Goods; of the
former sort, Bread, Beef, Pork, Pilchard, Herring, Cheese, Butter,
Salt, and Oil, are first in demand. The next are dry Goods, Hats,
Wigs, Shirts, Stockings, Kersys, Sagathys, Crapes, Says, Shalloons,
and Broadcloths, particularly Black Suits, the usual wear of the
Portuguese. The last and least in Expence are Escrutores, Chairs,
Pewter, Post-Paper, Counting-books, &c. For these you have in
Exchange their Wine at 30 Millrays a Pipe; the Malmsey, 60. each
Millray in present Pay 6s. 8d. in Bills 6s. What other little Traffick I
had, stands as per Margin.[4]
There is one Caution to be observed; That as there is not much
dishonour in Trade to take advantage of a Chapman’s Weakness, it
is prudent to see the Wines you have tasted shipped forthwith, or it is
odds but the Stranger finds them adulterated: So that altho’ they
seem to allow a good Interest on your Goods; yet the Badness of
your Wine, or (if good) broke at their Price, lessens the supposed
Advantage. Some Goods at particular times, bear an extraordinary
Price; not so much by a Call of the Island, as of Brasil, whither they
are again exported.
Fonchial is the chief Town of the Island, the Residence of the
Governour and Bishop: Is large and populous, has five or six
Churches; three Nunneries, not so strict as at Lisbon, we conversing
and trading for Toys with them every day; and as many Convents of
Fathers. That of the Jesuits has at present in it only seventeen; a
neat handsome Building and Chappel: this Order being in all
Catholick Countries the most respected for their Learning and
Riches. Wherever you find a College of them, you may be sure there
is good Living. The other Inhabitants consist of a mixed Race;
Portuguese, Blacks, and Molattoes, who are civil, courteous, and
equally respected in Trade; the Portuguese no where abroad
scrupling an Alliance with darker Colours.
They keep no regular Market, but the Country brings in according
as they think will be the Demand at any time: Kid, Pork, and now and
then a lean Heifer, Cabbages, Lemons, Oranges, Walnuts, Figs,
Yams, Bananoes, &c. There is one Curiosity I found in their Gardens
called the everlasting Flower, never fading after gathered, or
indiscernibly, in many Years; the Herb is like Sage growing, and the
Flower like Camomil: I laid by several of them, and found at twelve
Months end they were just of the same freshness as when gathered.
Fonchial Road is very open and unsafe against West and S. W.
Winds; deep Water also, that there is no anchoring but at the West
End, and that in 40 Fathom, a Mile or Mile and half off Shore: So that
when a Swell from those Quarters gives notice of a Gale coming, all
Ships in the Road slip their Cables and to Sea, returning at a more
favourable season for their lading: Which likewise, by an
extraordinary Surf on the Beach, becomes troublesome to ship off;
commonly done by swimming the Pipes off to the Lanch, or lade on
the Beach, and run her with many hands into the Sea. The like
trouble Boats have in Watering (by a River at the W. End of the
Town) and is most commodiously done before the Sea-breeze
comes in.
The Loo makes a tolerable Harbour for small Vessels against
Westerly Winds, that would be unsafe without. They make fast their
Cable to a high Rock called the Loo, whereon is a Fort; but when the
Winds veer, opening their Heads to the Sea, all Hands go on Shore,
and leave the Ship and Storm to contest it by themselves.
Their Lodgings on shore are as uneasy to Strangers, as the Road
to Ships; being prodigiously pestered with Bugs and Fleas. Cotts
upon the Floors, is the common way of laying.
Their Strength is in the Militia, computed at 18000 disciplin’d and
loyal Fellows; They, the Azores, and Cape De Verd Islands soon
returning to their Allegiance, after that Revolution in Portugal, 1640.
Before I leave Madeira, I must relate the surprizing Account just
arrived here by several Masters of Vessels, Eye-witnesses of a new
Island which sprung out of the Sea the 20th of November last, 17
Leagues S. E. from Terceira, one of the Western Islands.
The Master who took a Survey of it by order from the Governour of
Terceira, lays it down, a League long, a Mile broad, a little above the
surface of the Water, and smoking like a Volcano. After the Eruption,
the Sea for several Leagues round was covered with Pumice-stone,
and half-broiled Fish. I was curious to know what Symptoms (if any)
had preceded this Prodigy at the other Islands; and learned that
Pico, one of them, a noted Volcano, had ceased to burn for some
time, and that they had felt a Shock or two of an Earthquake that had
done considerable damage. Corvo, an Island in this Neighbourhood
([5]Albert de Mandelzo tells us) started up also in such manner, June
16, 1628. And History relates the like in the Archipelago.
That new Islands should be formed in Rivers, as at the Conflux of
the Save with the Danube, or Sands shifting in any Channels, may
be from the Swiftness of the Streams, wasting some and raising
others; but that this Effect should happen in deep Water, 50 or 60
Miles from Shore, is truly wonderful: The Phænomenon seems best
resolved here, by subterranean Fires, which from a great Depth and
Extent have their Vents at Volcanoes; and as the Consumption of
their Materials is more, the nigher they are such Vents (observable in
Italy, Iceland, &c.) so their Effects in the neighbourhood of Waters
(when by any Accident the Mouth is stopp’d, and they meet) must be
Concussions of the Earth, blowing the Mountains away in Cinders;
and now and then in Ages, such a Wonder as a new Island, the
same as we see (if we may compare great things with small) in
several Chymical Preparations. This Island has settled, and probably
by the Spunginess of its Materials, may sink in a few Years out of
sight again. The ultimate End, is perhaps to strike Mankind with a
Dread of Providence, and warn a sinful World against the
Consequences of angry Omnipotence: Men generally taking a
deeper Impression from something new and wonderful in Nature,
than in the Creation or Conservation of the World it self.
 CANARIES.
From Madeira we sailed by the Canary Islands, belonging to the
Spaniards, and taken by them in 1418.
Palma, remarkable for rich Wines, making 12000 Pipes per
Annum.
Ferro, or Ferrara, for our Navigators taking their first Meridian from
thence, there being none, or the least Variation; and for a Volcano
that now and then breaks out upon it. One in November 1677, seen
five Days; and in 1692 broke out again with Earthquakes, and seen
six Weeks together: There is also, our Voyages say, a wonderful
Tree on it, forty Foot high, that condenses the Clouds in such
quantity, as to supply the want of Springs.
Grand Canary, the chief Residence for Governours and Consuls;
and Teneriff, for its noted Pike, thought from the shewing it self
singly, to be the highest Land in the World. It is a Pyramidal Heap of
rough Rocks piled thus (it’s thought by Naturalists) from some
subterraneous Conflagration that burst out heretofore.
The Ancients called them Insulæ fortunatæ; it’s likely from the
Interception they may have given to the Destruction of Coasters
blown off, before the use of the Compass: Cape Non on the
Continent being the utmost of their Navigation.
 Cape de V E R D Islands,
Denominated from the Cape, always green: They were anciently
called Hesperides; the Diminutive of Spain, called heretofore
Hesperia, propterea quod hæc regio, omnium extrema, sit a sit ad
Occidentem; Hesperus, the Evening Star, by a Metaphor signifying
the West.
They are inhabited by Portuguese, who welcome all sort of Ships
(of good, or ill Design) bound to Guinea, India, Brasil, or the West-
Indies; they frequently putting in here to furnish themselves with
fresh Provisions, exchanged for Trifles; chiefly at St. Iägo (James)
the principal, which has three or four Forts, and where resides the
Governour. In several of these Islands there are natural Salt Ponds,
kerning great quantities without trouble. The most noted by the
English is Maio, or the Isle of May, where many of our Ships lade in
Summer; and was, with Tangier, and Bombay in India, Part of Q.
Catherine’s Fortune to England. Another of them has a Volcano, and
called Del Fuego.
The Land about the Cape appears the Height of that at Deal in
Kent; woody, a white even Sand along Shore, and about 28 Fathom
Water a League off. Just to the Northward are two or three great
Rocks, called by our Sailors Shitten Islands, being white all over with
the Dung of Sea-fowls. At the same distance Southward of the Cape,
is an Island called Goree, about a League from the Main, has a
French Factory with two Forts, commanding all the Trade about the
River Senega, from other Nations.
While our Ships lay too here, we had good Fishing with our Lines;
took Breams (or Porga’s) Skip-jacks, Groupes, a Rock-fish (thick,
short, and of a deep yellow on the Belly, Gills, and Mouth) and the
Jew-Fish; which has a double Mouth, the uppermost not to swallow
Food, but full of Air-pipes, and finned like a Cod, all well tasted: and
having washed them down with a Bowl, our Friends and we parted,
the Weymouth steering in for Gambia River with the Governour
Colonel Witney, and the Merchants; We for Sierraleon, anchoring
there the Beginning of April, 1721.
The Winds from Madeira to Sierraleon at first blew fresh at S. and
S. W. and as we came farther to the Southward, they wheel’d
gradually on the Western Side of the Circle, quite round to the N. so
as in the Latitude of 21 to have it N. E. a true Trade, seven, eight, or
nine knots Day and Night; but whether it were the Badness of our
half-minute Glasses, the tendency of the Sea with the Wind, or any
Current, I cannot tell; but we always found our selves considerably
further to the Southward, by Observation every day, than the
Distance by the Log would give.
In this Passage, we took up a few Turtle with our Boat. As they
sleep and bask upon the Surface, we steal upon them without noise,
and throw them in upon their Backs. We saw also abundance of
flying Fish, and their continual Enemies, the Albicore and Dolphin;
the latter we strike now and then with a Fizgig, or Harping-iron. It is a
glorious-colour’d, strait Fish, four or five Foot long, forked Tail,
perpendicular to the Horizon: plays familiarly about Ships; is of dry
Taste, but makes good Broth. They are seldom seen out of the
Latitudes of a Trade-wind; and the flying Fish never: These are the
bigness of small Herrings; their Wings about two thirds its length;
come narrow from the Body, and end broad; they fly by the help of
them a Furlong at a time when pursued, turning in their Flight,
sometimes dip in the Sea, and so up again, the Wind making them,
by this Expedient, fleeter.
 A F R I C A in general.
As there is nothing more surprizing and delightful in Voyages or
Travels, than beholding the different Habits, Customs, Dieting, and
Religion of the different Natives; so there is none I believe, wherein
that Difference can be found, so much as here. A Colour, Language
and Manners, as wide from ours, as we may imagine we should find
in the planetary Subjects above, could we get there.
But before I proceed on any Observations of my own, it may be
proper from others, to convey some Idea of Guinea in general; so
much as carries Probability, either from the Dead or Living.
Africa, one of the four Quarters of the World, next in bigness to
Europe, by the Ancients had several Names; Olympia, Ammonis
Ortygia; but the most noted, Apher, from a Nephew, it’s said, of
Abraham’s. It extends from about 36 N. to as many Degrees of
Southern Latitude; and excepting Egypt, Barbary, Morocco, and in
this last Age the Coast of Guinea, is a Country as little known as any
Part of the Globe. Marmol says, the Arabians in the 400 of the
Hegyra, passed into Afric and divided it. This is certain, that it has
many fine large Rivers, some of them navigable for Ships. Along the
Banks of these Rivers, the Inhabitants abound with Millet, Rice,
Pulse or, Indian-Corn. The further we depart from Morocco on this
West Side, or Egypt on the East, there is always found less Industry
and more Ignorance: For Governments, tho’ never so tyrannical, are
better than none, extending some Improvement to Humanity.
The Niger, which is one of the largest Rivers in Africa, is said to
have the same Property of overflowing every Year, like Nile,
remunerating to the inland parts a vast Fertility and Increase; and
this very probably, because it has been traced some hundred
Leagues, and by the Course, descends from the Ethiopian
Mountains, the common Fountain of both.
 The Senega and Gambia, Branches of this great River, disgorge
here at the windward Part of Guinea; they are large Rivers, driving
considerable Trade: To the former of these, the King of Morocco
extended his Dominions, about 1526, by the Conquest of the
Kingdom of Tombuto, which still continues tributary, and whence that
King raises considerable Negro Armies, his chief Strength. A College
of the Sect of Haly, is founded in Melli, a Kingdom upon this River.
They have many Crocodiles or Alligators, Sea-Horses, and Shirks in
them. Senega affords great quantity of Gum; and at Gambia begin
our Factories for Slaves, Teeth, and Gold, on which this general
Remark, That the Slaves there, faring softer from a better Soil, are
not so hardy as those lower down. The Teeth are as large, and in as
much plenty, as at any one Part of the whole Coast; those taken out
of the Sea-Horse are small, not weighing above 5 or 6 Pounds, but
more solid than the Elephant’s. And lastly, their Gold is current in
what the Traders call Bars, little twisted Lengths, or in Rings of 4, 5,
6, 7, or 8s. Value.
All the great Rivers flow and ebb regularly, being governed by the
Moon, as the Tides on our own Coasts; but the Sandiness of the
Soil, and Nearness of the Sun, makes the Country between, so
extreamly dry, that they have great scarcity of Water for an hundred
Miles an end sometimes; and this Drought is what brings the Beasts
of all sorts in Droves to the Banks, for satisfying Thirst, (Tygers,
Panthers, Leopards, Antelopes, Elephants, Apes; Ostriches, &c.)
From which Accident, say they, might probably have happened the
many Hebridous Productions that have made this Country the
Proverb of all Ages; it continually producing something new or
monstrous.
Their chief Diet is Indian Corn, Rice, Palm-nuts, Bananas, Yamms,
Pine-apples, and now and then a little Fish, or a Fowl; all which thro’
Ignorance, and want of Necessaries, are very slovenly cooked by
them.
Africa is almost a Triangle in shape; the Kingdoms on the North
are Mahometans; and in the trading Towns of Barbary, and Turky,
there is a little Mixture of Jews. On the Eastern Line next Persia, are
said to be some of the Sect of Gaurs, followers of Zoroastes, a very
learned Persian Philosopher, that appeared, according to Dr.
Prideaux, about 2300 Years ago: He instituted Fire-worship, and
established it by a superiour Cunning, through most parts of Persia
and India, where there are still some left, poor and despised, (called
Persees) since the seventh Century, when the Mahometans over-run
that Country, and almost extinguished them. In Æthiopia, (Prester
John’s Country) Writers say, are a sort of Christians, still
acknowledging the Patriarch of Alexandria; meerly nominal I believe,
for the Greeks themselves, much nigher his Pastorship, have since
their Conquest by the Turks, in a manner lost their Christianity;
Poverty and Ignorance, the Consequence of Captivity having
obliterated the outward Pomp, which, next to Power, is the main
Pillar in all Religions. Inland, and to the Southern Extremity, they are
Pagans. And on this Western Line (the Negroes) all trust to the
Gregory or Fetish; which in the bulk of it means no more than what
we in Europe call Charms, which in many respects carries strong
Superstition, that is, a vain Religion in it; only their consecrated
Materials having more Reverence from their Ignorance and Fear,
work more stupendous Effects; or are imagined to do so, which is
the same thing. So much may serve for a general Idea of Africa,
since several of the Articles will, in the progress of the Voyage, be
occasionally expatiated on.
 SIERRALEON.
By Guinea here, I mean all Negro-land, from about the River Senega
Northward, to within a few Degrees of Cape Bon Esperance;
because Ships bound to any part of this Extent, are said to be bound
to Guinea; and because the People, without these Lines, alter to a
dark Colour seen in the Moors at this, and the Hottentots at the other
Extremity. The Name (Gordon says) imports hot and dry, and its
Gold gives Name to our Coin.
The black Colour, and woolly Tegument of these Guineans; is what
first obtrudes it self on our Observation, and distinguishes them from
the rest of Mankind, who no where else, in the warmest Latitudes,
are seen thus totally changed; nor removing, will they ever alter,
without mixing in Generation. I have taken notice in my Navy-
Surgeon, how difficultly the Colour is accounted for; and tho’ it be a
little Heterodox, I am persuaded the black and white Race have, ab
origine, sprung from different-coloured first Parents.
When we parted with the Weymouth off Cape de Verd, we steered
S. S. W. to avoid the Shoals of Grandee, and in hawling in for the
Land again, waited till we came into the Latitude of Sierraleon, some
others laying on the N. Side that River. The Soundings in with the
Cape are gradual, from 60 Fathoms about 12 Leagues off, to 13;
when we get in sight of Cape Sierraleon, known by a single Tree
much larger than the rest, and high land on the back of it. We run up
on the Starboard side of the River, anchoring in the third Bay from
the Cape; where is very commodious watering and wooding; and
regular Tides, as in any part of the Channel of England.
Remark 1. The Trade for our African Company here, is carried on
from Bense or Brent Island, about 5 Leagues distance from our
Anchorage, by Factors, of whom Mr. Plunket is chief. The private
Traders are about 30 in number, settled on the Starboard side of the
River: loose privateering Blades, that if they cannot trade fairly with
the Natives, will rob; but then don’t do it so much in pursuance of
that trading Advice, (Amass Riches, my Son,) as to put themselves
in a Capacity of living well, and treating their Friends, being always
well pleased if they can keep their Stock at Par, and with their Profits
purchase from time to time, Strong-beer, Wine, Cyder, and such
Necessaries, of Bristol Ships, that more frequently than others put in
there; of these, John Leadstine, commonly called old Cracker, is
reckoned the most thriving.
They all keep Gromettas (Negro Servants) which they hire from
Sherbro River, at two Accys or Bars a Month. The Women keep
House, and are obedient to any Prostitutions their Masters
command. The Men-servants work in the Boats and Periagoes,
which go a trading in turns with Coral, Brass, Pewter Pans, Pots,
Arms, English Spirits, &c. and bring back from the Rio Nunes,
Slaves, and Teeth; and from Sherbro, Camwood for Dyers; a Sloop
or two is the most that is loaded from the latter Place in a Year, and
that with difficulty; being obliged to go far up the River, narrow and
beset with Mangroves, which makes it sickly.
The Ivory here is of the Elephant or Sea-Horse, great and small;
the former, sold at about 40 Accys per Quintal in Exchange; the
other at half Price.
The Slaves when brought here, have Chains put on, three or four
linked together, under the Care of their Gromettas, till Opportunity of
Sale; and then go at about 15 Pounds a good Slave, allowing the
Buyer 40 or 50 per Ct. Advance on his Goods.
As these Slaves are placed under Lodges near the Owner’s
House, for Air, Cleanliness, and Customers better viewing them, I
had every day the Curiosity of observing their Behaviour, which with
most of them was very dejected. Once, on looking over some of old
Cracker’s Slaves, I could not help taking notice of one Fellow among
the rest, of a tall, strong Make, and bold, stern aspect. As he
imagined we were viewing them with a design to buy, he seemed to
disdain his Fellow-Slaves for their Readiness to be examined, and
as it were scorned looking at us, refusing to rise or stretch out his
Limbs, as the Master commanded; which got him an unmerciful
Whipping from Cracker’s own Hand, with a cutting Manatea Strap,
and had certainly killed him but for the loss he himself must sustain
by it; all which the Negro bore with Magnanimity, shrinking very little,
and shedding a Tear or two, which he endeavoured to hide as tho’
ashamed of. All the Company grew curious at his Courage, and
wanted to know of Cracker, how he came by him; who told us, that
this same Fellow, called Captain Tomba, was a Leader of some
Country Villages that opposed them, and their Trade, at the River
Nunes; killing our Friends there, and firing their Cottages. The
Sufferers this way, by the Help of my Men, (says Cracker) surprized,
and bound him in the Night, about a Month ago, he having killed two
in his Defence, before they could secure him, and from thence he
was brought hither, and made my Property.
Remark 2. Sierraleon River is very broad here, but in ten or twelve
Miles rowing upwards, narrow to half the Breadth of the Thames at
London, spread on both sides thick with Mangroves; Trees, or
slender woody Shrubs, that spring from the low, watry Banks of
Rivers, in warm Climates. From the Branches, the Sap descends
again and takes a second Root, and so on, a third, fourth, &c. that
the Ground is all covered; very difficult, if not impossible for Men to
penetrate: This makes them fit Haunts for the Manatea and
Crocodile (Sea-Cow and Alligator) which, with the Shirks, very much
infest the River. A Story or two of these Creatures, may not be
unacceptable.
The Manatea is about eleven or twelve Foot long, and in girt half
as much; Teeth only in the back part of her Mouth, which are like the
Ox’s, as is also her Muzzle and Head; with this difference, that her
Eyes are small in proportion, and Ears you can scarce thrust a
Bodkin in; close to her Ears almost, are two broad Finns, sixteen or
eighteen Inches long, that feel at the Extremities as tho’ jointed; a
broad Tail, Cuticle granulated, and of a colour and touch like Velvet:
the true Skin an Inch thick, used by the West-Indians in Thongs for
punishing their Slaves; weigh to five or fix hundred Weight; of a firm
Flesh, that cuts fat, lean, and white like Veal: Boiled, stewed, or
roasted (for I have eaten it all ways) it has no fishy Taste, but is as
acceptable a Treat as Venison to Cockneighs.
The Negroes way of taking them, is in a Canoo, which they paddle
towards the Manatea with as little noise as possible, (she being
extreamly quick of hearing:) when near enough, a Man placed ready
in the Boat’s Head, strikes in his Harpoon with a long Pole into her,
and lets go. She makes towards the Mangroves immediately, and
the Water being shallow, they now and then get sight of the Pole,
and so follow, renewing the Strokes till they kill, or weary her, and
then drag her ashore.
The Alligator answers in all respects, and doubtless is the
Egyptian Crocodile; shaped not unlike the Lizard, but of two hundred
Weight perhaps, covered with hard Scales that are impenetrable to
Shot, unless very near; long Jaws set with sharp Teeth, two very
large, and two small: Finns like Hands: A Tail thick and continuous;
will live a long time out of the Water, being sold frequently alive in the
West-Indies. They are not shy, but rather bold, and tho’ easily
waked, will not make off presently, our Boats falling down with the
Stream within a few yards of them, before they stir; laying basking to
the Sun, in little muddy Nooks they form in their egress from the
Mangroves. When they float upon the Water, they lie very still and
like a Log of Timber, till the little Fry underneath come unwarily
sporting about them and tempt their greedy Stomachs; they diving
very quick upon their Prey.
One of these set upon a Man of Captain Masterton’s, a Sloop that
put in here from Sherbro. The Sailor, to avoid walking round a Bay,
and being mellow with drinking, would needs cut his way short by
wading over a weedy part of it up to his Breast, where the Alligator
seized him; and the Fellow having full Courage, ran his Arm down
his Throat: Notwithstanding which, the Crocodile loosed, and
renewed the Battle two or three times, till a Canoo that saw the
Distress, paddled to his Relief, but he was torn unmercifully in his
Buttocks, Arms, Shoulders, Thighs, and Sides; and had not the
Creature been young, must certainly have been killed. The Man
recovered of his Wounds.
 Shirks very much infest the Mouth of this River; the most bold and
ravenous of the watry Tribe: He never forsakes your Hook, till he is
taken, and slights the Proverb,
Occultum visus decurrere piscis ad hamum.
We have catched three in less than half an hour, each 8 or 10 Foot
long, the Livers of them making above ten Gallons of Oil. They have
four or five Rows of short, sharp Teeth, one within another, and the
Sides of them indented like Saws. Their Swallows 14 and 16 Inches
over. In the Maws, we found Beef Bones, and what other Trash had
been thrown over-board in the Day; for they are like the Parson’s
Barn; they turn on their backs to take in their Prey. Our Seamen
dressed and eat the Flesh, tho’ very strong; the fault of all
carnivorous Animals.
These Shirks have generally two, three, or more pretty-coloured
little Fish, the bigness of Herrings attending them, called Pilots: They
go in and out at his Maw, or fasten on his Back, in familiar manner:
They are supposed like the Jackall to the Lion, to be instrumental in
procuring him Prey, and warning him of Danger in Shoals, for which
he receives Food, and Protection from the Shirk.
I shall give an Instance or two within my own knowledge, to shew
the Boldness and Rapacity of this Fish.
The Weymouth’s Barge rowing up Gambia River, a Shirk made to
them, and notwithstanding the noise of so many Oars, seized one of
them in his Mouth, and snapped it in two.
At Whydah, a very dangerous Coast to land at, having two Bars
before it, and great Seas; a Canoo was going on shore from a
Merchant-Ship with some Goods, and in attempting to land, overset:
A Shirk nigh hand, seized upon one of the Men in the Water, and by
the Swell of the Sea, they were both cast on shore; notwithstanding
which, the Shirk never quitted his hold, but with the next Ascend of
the Sea, carried him clear off.
In short, their Voracity refuses nothing; Canvas, Ropeyarns,
Bones, Blanketing, &c. I have seen them frequently seize a Corpse,
as soon as it was committed to the Sea; tearing and devouring that,
and the Hammock that shrouded it, without suffering it once to sink,
tho’ a great Weight of Ballast in it.
There are in the Bays of this River, variety of good Fish, that
supplies the Scarcity of Flesh; Turtle, Mullet, Skate, [6]Ten-pounders,
[7]
Old-wives, [8]Cavalloes, [9]Barricudoes, [10]Sucking-Fish, Oysters,
[11]
Cat-Fish, Bream, and Numb-Fish; the most of which we catch’d in
great numbers with our Searn; two or three Hours in a Morning
supplying a Belly-full to the whole Ship’s Company.
The Oysters and Numb-Fish have something peculiar; the former
growing, or rather sticking in great Bunches of twenty or thirty, upon
the Rocks and Mangroves, to which they seemingly grow, very small
and ill-tasted.
The latter, which is the Torpedo of the Ancients, is flat as a Skate,
so very cold as to numb the Hands or Arms of those who touch him,
but goes off again in few Hours; and with a Stick you may toss him
about a Day together without any other Harm than losing your time.
Remark 3. The Country about Sierraleon is so thick spread with
Wood, that you cannot penetrate a Pole’s length from the Water-
side, unless between the Town, and Fountain whence they fetch
their Water, without a great deal of difficulty. They have Paths
however through these Woods, to their [12]Lollas, and [13]Lugars,
which tho’ but a mile or two from the Town, are frequently the Walks
of wild Beasts; their Excrement I have found up and down in walking
here, white and mixed with Ossicles.
The Shores hereabouts, like those of Sweden, are rocky, and
without any Cover of Earth almost; yet produce large Trees, the
Roots spreading on the Surface: The chief of these are the Palm, the
Coco, and the Cotton-Tree, described p. 198, in the first Volume of
the History of the Pyrates.
Other Vegetables for Food are Rice, Yams, [14]Plantanes, [15]Pine-
Apples, [16]Limes, Oranges, [17]Papais, Palm-nuts, wild Roots, and
Berries.
This is their common Sustenance; the Gift of Providence, without
their Care; they might abound, but prefer Ease and Indolence, he