Test Methodology: Enterprise Firewall

TEST METHODOLOGY
Enterprise Firewall
February 5, 2021
V1.0
CyberRatings.org Test Methodology
Table of Contents
Introduction ......................................................................................................................................... 5
1.1 Enterprise Firewalls .......................................................................................................................5
1.2 About This Test Methodology ........................................................................................................5
1.3 Inclusion Criteria ............................................................................................................................5
1.4 Product Guidance ..........................................................................................................................6
2 Access Control ............................................................................................................................... 7

2.1 Baseline Policy ...............................................................................................................................7
2.2 Simple Policy .................................................................................................................................7
2.3 Complex Policies with Application Control .....................................................................................7
2.3.1 Block Unauthorized Applications ............................................................................................................................ 7
2.3.2 Block Specific Action (Depends on the application) ................................................................................................ 7
3 SSL/TLS Functionality .................................................................................................................... 8

3.1 Decryption Validation ....................................................................................................................8
3.1.1 Office 365 Support .................................................................................................................................................. 8
3.2 Cipher Selection .............................................................................................................................8

3.3 Cipher Support...............................................................................................................................8
3.3.1 Top 24 Cipher Suites from the Tranco Top 1 Million, as of March 2020:................................................................ 8
3.3.2 Prevention of Weak Ciphers ................................................................................................................................... 9
3.3.3 Support for TLS 1.3 .................................................................................................................................................. 9
3.4 Decryption Bypass Exceptions ........................................................................................................9

3.5 Certificate Validation ................................................................................................................... 10
3.6 TLS Session Re-use ....................................................................................................................... 10
4 Threat Prevention ....................................................................................................................... 11

4.1 False Positives ............................................................................................................................. 11
4.1.1 Initial check – legitimate traffic, documents, and files ......................................................................................... 11
4.1.2 Ongoing check – legitimate traffic, documents, and files ..................................................................................... 11
4.2 Exploits ........................................................................................................................................ 11

4.3 Protection Resiliency ................................................................................................................... 12
4.4 Evasions....................................................................................................................................... 15
4.4.1 IP Packet Fragmentation ....................................................................................................................................... 15
4.4.2 TCP Stream Segmentation .................................................................................................................................... 15
4.4.3 HTTP Obfuscation.................................................................................................................................................. 16
4.4.4 HTTP Compression ................................................................................................................................................ 17
4.4.5 HTML Obfuscation ................................................................................................................................................ 18
5 Performance ............................................................................................................................... 19
5.1 Raw Packet Processing Performance (UDP Throughput)............................................................... 19
5.1.1 64 Byte Packets ..................................................................................................................................................... 19
5.1.2 128 Byte Packets ................................................................................................................................................... 19
Enterprise Firewall v1.0

2
5.1.3 256 Byte Packets ................................................................................................................................................... 19

5.1.4 512 Byte Packets ................................................................................................................................................... 19
5.1.5 1024 Byte Packets ................................................................................................................................................. 19
5.1.6 1514 Byte Packets ................................................................................................................................................. 19
5.2 Latency ........................................................................................................................................ 20

5.2.1 64 Byte Frames ..................................................................................................................................................... 20
5.2.2 128 Byte Frames ................................................................................................................................................... 20
5.2.3 256 Byte Packets ................................................................................................................................................... 20
5.2.4 512 Byte Packets ................................................................................................................................................... 20
5.2.5 1,024 Byte Packets ................................................................................................................................................ 20
5.2.6 1,514 Byte Packets ................................................................................................................................................ 20
5.3 Maximum Capacity ...................................................................................................................... 20

5.3.1 Theoretical Maximum Concurrent TCP Connections ............................................................................................ 20
5.3.2 Maximum TCP Connections per Second ............................................................................................................... 21
5.3.3 Maximum HTTP Connections per Second ............................................................................................................. 21
5.3.4 Maximum HTTP Transactions per Second............................................................................................................. 21
5.3.5 Maximum HTTP(S) Connections per Second ......................................................................................................... 21
5.4 HTTP Capacity .............................................................................................................................. 21

5.4.1 44 KB HTTP Response Size – 2,500 Connections per Second ................................................................................ 22
5.4.2 21 KB HTTP Response Size – 5,000 Connections per Second ................................................................................ 22
5.4.3 10 KB HTTP Response Size – 10,000 Connections per Second .............................................................................. 22
5.4.4 4.5 KB HTTP Response Size – 20,000 Connections per Second ............................................................................. 22
5.4.5 1.7 KB HTTP Response Size – 40,000 Connections per Second ............................................................................. 22
5.5 Application Average Response Time: HTTP .................................................................................. 22

5.6 HTTPS Capacity ............................................................................................................................ 23
5.6.1 1.7 KB HTTPS Response Size – 40,000 Connections per Second ........................................................................... 23
5.6.2 4.5 KB HTTPS Response Size – 20,000 Connections per Second ........................................................................... 23
5.6.3 10 KB HTTPS Response Size – 10,000 Connections per Second ............................................................................ 23
5.6.4 21 KB HTTPS Response Size – 5,000 Connections per Second .............................................................................. 23
5.6.5 44 KB HTTPS Response Size – 2,500 Connections per Second .............................................................................. 23
5.7 Application Average Response Time: HTTPS................................................................................. 23

5.8 "Real-World" Single Application Flows......................................................................................... 24
6 Stability and Reliability ............................................................................................................... 25

6.1 Blocking Under Extended Attack .................................................................................................. 25
6.2 Passing Legitimate Traffic under Extended Attack ........................................................................ 25
6.3 Behavior of the State Engine Under Load ..................................................................................... 25
6.3.1 Attack Detection/Blocking – Normal Load ............................................................................................................ 25
6.3.2 State Preservation – Normal Load ........................................................................................................................ 26
6.3.3 Pass Legitimate Traffic – Normal Load .................................................................................................................. 26
6.3.4 State Preservation – Maximum Exceeded ............................................................................................................ 26
6.3.5 Drop Legitimate Traffic – Maximum Exceeded ..................................................................................................... 26
6.4 Protocol Fuzzing & Mutation........................................................................................................ 26

6.5 Power Fail .................................................................................................................................... 26
6.6 Backup/Restore ........................................................................................................................... 26
6.7 Persistence of Data ...................................................................................................................... 26

3
7 Management & Reporting Capabilities ....................................................................................... 27

7.1 Authentication ............................................................................................................................. 27
7.1.1 Role-Based Access Control (RBAC) ........................................................................................................................ 27
7.1.2 Authentication ...................................................................................................................................................... 27
7.2 Policy ........................................................................................................................................... 27

7.2.1 Policy Definition .................................................................................................................................................... 27
7.2.2 View Policy ............................................................................................................................................................ 27
7.2.3 Policy Association.................................................................................................................................................. 27
7.2.4 Policy Inheritance.................................................................................................................................................. 27
7.2.5 Policy Version and Checksums .............................................................................................................................. 27
7.2.6 Bulk Operations..................................................................................................................................................... 27
7.3 Logging ........................................................................................................................................ 27

7.3.1 Malicious Traffic .................................................................................................................................................... 28
7.3.2 Administrator Login/Logout .................................................................................................................................. 28
7.3.3 Successful Authentication ..................................................................................................................................... 28
7.3.4 Unsuccessful Authentication................................................................................................................................. 28
7.3.5 Policy Change ........................................................................................................................................................ 28
7.3.6 Policy Deployed ..................................................................................................................................................... 28
7.3.7 Hardware Failure................................................................................................................................................... 28
7.3.8 Power Cycle........................................................................................................................................................... 28
7.3.9 Log Time Normalization ........................................................................................................................................ 28
7.3.10 Log File Maintenance ............................................................................................................................................ 28
7.3.11 Forensic Analysis ................................................................................................................................................... 28
7.4 Alert Handling.............................................................................................................................. 28

7.4.1 Centralized Alerts .................................................................................................................................................. 28
7.4.2 Performance Alerting ............................................................................................................................................ 28
7.4.3 Alert Filtering ........................................................................................................................................................ 28
7.4.4 View Alert Detail ................................................................................................................................................... 29
7.4.5 View Packet Contents ........................................................................................................................................... 29
7.4.6 Alert Suppression .................................................................................................................................................. 29
7.4.7 Incident Workflow ................................................................................................................................................ 29
7.4.8 Correlation (automatic)......................................................................................................................................... 29
7.4.9 Correlation (manual) ............................................................................................................................................. 29
7.5 Reporting ..................................................................................................................................... 29

7.5.1 Custom Reports..................................................................................................................................................... 29
7.5.2 Saved Reports ....................................................................................................................................................... 29
7.5.3 Report Automation ............................................................................................................................................... 29
7.5.4 Centralized Reports ............................................................................................................................................... 29
7.5.5 Built-In Reports ..................................................................................................................................................... 29
7.5.6 Industry Reporting Standards ............................................................................................................................... 29
7.6 Change Control ............................................................................................................................ 29

7.6.1 Change Control Logging ........................................................................................................................................ 30
7.6.2 Roll-Back................................................................................................................................................................ 30
7.6.3 Revision History .................................................................................................................................................... 30
8 Total Cost of Ownership and Value.............................................................................................. 31

Contact Information ........................................................................................................................... 32

4
Introduction
1.1 Enterprise Firewalls
The firewall market is one of the largest and most mature security technology segments. A firewall is a mechanism
used to protect a trusted network from an untrusted network while allowing authorized communications to pass
from one side to the other, thus facilitating secure business use of the Internet.
Firewalls have undergone several stages of development, from early packet filtering and circuit relay firewalls to
application layer (proxy-based) and dynamic packet filtering firewalls. Throughout their history the goal has been to
enforce an access control policy between two networks, and they should therefore be viewed as an implementation
of policy.
The Enterprise Firewall must be capable of performing deep packet inspection (DPI) on all packets, on all ports, and
over all protocols to determine which applications are running over which ports and thus secure them effectively. In
addition, with the expanded use of SSL/TLS in much of the traffic traversing the modern network, inspection of
encrypted content is required.
1.2 About This Test Methodology

CyberRatings.org test reports are designed to address the challenges faced by enterprise security and IT
professionals in selecting and managing security products. The scope of this particular methodology includes:
• Security effectiveness • Stability and reliability

• Performance • Total cost of ownership (TCO)
As firewalls are deployed at critical choke points in the network, their stability and reliability are imperative.
Therefore, regardless of any new deep inspection capabilities, the main requirement of any firewall is that it must
be as stable, as reliable, as fast, and as flexible as the firewall that it is replacing.
The following capabilities are considered essential in a firewall:
• Packet filtering • Integrated intrusion prevention system (IPS)

• Stateful inspection through OSI Layer 3 • External intelligence to enhance blocking
• Network Address Translation (NAT) decisions (i.e., "reputation services")
• Stable, reliable, low latency • Anti-malware
• Application awareness/control • SSL decryption
1.3 Inclusion Criteria

To encourage the greatest participation and allay any concerns of bias, CyberRatings invites all leading Firewall
vendors to submit their offering at no cost. Vendors with significant market share, as well as challengers with
innovative technology, will be included at CyberRatings discretion.
CyberRatings determines a vendor's inclusion in a group test based on an analysis of the market and an
understanding of the criteria important to our customers. Some of the elements considered are:
• Market presence
• Identified by industry analysts covering the specific technology area
• Consumer requests
• Innovative technology/solution (requires internal vetting for emerging vendors)

5
1.4 Product Guidance

CyberRatings issues summary product guidance based on evaluation criteria that is important to information
security professionals. The evaluation criteria are as follows:
• Security effectiveness — How effectively does the firewall protect control network access, applications, and
users while preventing threats (exploits, malware, phishing, etc.)?
• Resistance to evasion — Failure in any evasion class permits attackers to circumvent protection.
• Stability — Long-term stability is particularly important for a firewall offering, where failure can produce crippling
network outages.
• Performance — If a firewall offering slows down users, it will never be implemented, or those users will make
the operations team miserable.
• Management — In particular, how difficult is it to configure, maintain, and operate (i.e., find information)?
• Value — Customers should seek appropriate TCO and high effectiveness and performance rankings.
PRODUCT RATINGS
RATING DEFINITION
A product rated 'AAA' has the highest rating assigned by CyberRatings. The product's capacity to meet its commitments
AAA
to consumers is extremely strong.
A product rated 'AA' differs from the highest-rated products only to a small degree. The product's capacity to meet its
AA
commitments to consumers is very strong.
A product rated 'A' is somewhat less capable than higher-rated categories. However, the product's capacity to meet its
A
commitments to consumers is still strong.
A product rated 'BBB' exhibits adequate stability and reliability. However, previously unseen events and use cases are
BBB
more likely to negatively impact the product's capacity to meet its commitments to consumers.
A product rated 'BB,' 'B,' 'CCC,' 'CC,' and 'C' is regarded as having significant risk characteristics. 'BB' indicates the least
degree of risk and 'C' the highest. While such products will likely have some specialized capability and features, these
may be outweighed by large uncertainties or major exposure to adverse conditions.
A product rated 'BB' is more susceptible to failures than products that have received higher ratings. The product has
BB the capacity to meet its commitments to consumers. However, it faces minor technical limitations that have a potential
to be exposed to risks.
A product rated 'B' is more susceptible to failures than products rated 'BB'; however, it has the minimum capacity.
B Adverse conditions will likely expose the product's technical limitations that lead to an inability to meet its
commitments to consumers.
A product rated 'CCC' is susceptible to failures and is dependent upon favorable conditions to perform expected
CCC functions. In the event of adverse conditions, the product is not likely to have the capacity to meet its commitments to
consumers.
A product rated 'CC' is highly susceptible to failures. The 'CC' rating is used when a failure has not yet occurred, but
CC
CyberRatings considers it a virtual certainty.
A product rated 'C' is highly susceptible to failures. The product is expected to fail under any abnormal operating
C conditions and does not offer a useful management systems and logging information compared with products that are
rated higher.
A product rated 'D' is actively underperforming and failing and does not meet the use-case. The 'D' rating is used when
the product is not operational without a major technical overhaul. Unless CyberRatings believes that such technical
D
fixes will be made within a stated grace period (typically 30-90 calendar days), the 'D' rating also is an indicator that
existing customers using the product have already experienced a failure and should take immediate action.

6
2 Access Control
Firewalls must support stateful firewalling either by managing state tables to prevent "traffic leakage" or as a
stateful proxy. The firewall must be able to manage policies across multiple interfaces/zones. CyberRatings also
requires that a single security policy be applied to all interfaces under test. At a minimum, the firewall must provide
a "trusted" internal interface, an "untrusted" external/Internet interface, and (optionally) one or more DMZ
interfaces. In addition, a dedicated management interface (virtual or otherwise) is preferred.
This section verifies that the firewall is capable of enforcing a specified security policy
effectively. The test is conducted by incrementally building upon a baseline
configuration (simple configuration with no policy restrictions and no content
inspection) to a complex, real-world, multiple-zone configuration supporting many
addressing modes, policies, applications, and inspection engines.
At each level of complexity, test traffic is sent to ensure that only specified traffic is
allowed, and the rest denied, and those appropriate log entries are recorded.
2.1 Baseline Policy

A simple configuration containing an "allow all" policy. The purpose of the baseline
policy is to ensure traffic is flowing correctly through the device.
2.2 Simple Policy

• A "deny all" policy that blocks all unsolicited inbound connections.
• An outbound policy allowing basic internet services to be accessible by internal clients.
2.3 Complex Policies with Application Control

Complex outbound and inbound policies consist of many rules, objects, and applications, verifying that the firewall
can correctly determine the correct application from deep packet inspection (regardless of port/protocol used) and
taking the appropriate action.
• Popular social networking Web sites (Web applications)

• Messaging
• Video and Voice communications (e.g., WebEx Teams, Skype, Facetime, etc.)
• Business applications such as Salesforce, Netsuite, etc.
• Other popular applications TBD
For each application, CyberRatings will test the firewall's ability to perform the following functions
2.3.1 Block Unauthorized Applications
The firewall should be able to identify an unauthorized application and block it correctly
2.3.2 Block Specific Action (Depends on the application)
For example, with messaging, the firewall should allow text communications while blocking file transfers.

7
3 SSL/TLS Functionality
The use of the Secure Sockets Layer (SSL) protocol and its current iteration, Transport Layer Security (TLS), is rising
dramatically in response to an ever-increasing need for online privacy. In March 2020, data collected by Tranco on
their Top 1 Million1 showed that 60% of web traffic is being sent over HTTPS. While CyberRatings believes
encryption is a good thing, SSL/TLS is susceptible to various security attacks at multiple levels of network
communication. Attacks have been observed in the handshake protocol, record protocol, application data protocol,
and Public Key Infrastructure (PKI), to name just a few.
To address the growing threat of focused attacks using the most common web protocols and applications,
CyberRatings tests the capabilities of cloud network firewalls to provide visibility into the SSL/TLS payloads and
detect attacks concealed by encryption as well as attacks against the encryption protocols themselves. Performance
testing of SSL/TLS may be found in section 5.
3.1 Decryption Validation

To confirm that the firewall is correctly decrypting and (if applicable) inspecting SSL/TLS traffic, a validation test will
be performed before conducting functional or performance testing. The firewall offering will be expected to cover
all test cases in this methodology with a single configuration.
This test uses a known (previously blocked) exploit embedded in encrypted traffic and passed through the firewall.
The firewall will be expected to decrypt the stream, detect the exploit, and block or alert as appropriate. The
purpose of this test is not to evaluate the security effectiveness of the firewall but rather to validate that it is
appropriately decrypting and inspecting traffic.
3.1.1 Office 365 Support
Setup a deployment with a default "SSL inspect all traffic" rule and evaluate the process of getting a full O365
deployment working with it. Verify operation of desktop, mobile, and web applications without issue.
3.2 Cipher Selection

To determine the most commonly employed cipher suites for inclusion in testing, the top 30 ciphers were selected
from the March 2020 results of the Tranco Top 1 Million analysis for functional capability testing; and the top four
(representing more than 90% of the distribution) were selected for performance.
3.3 Cipher Support

The firewall offering is expected to negotiate a wide range of commonly used SSL/TLS ciphers to increase the
security visibility of potential threats encapsulated in real-world SSL/TLS traffic. This test will cover the top 30 cipher
suites as determined in section 3.3.1 of this methodology. Unless otherwise specified, the functional tests will use
the most common key sizes for RSA (2,048 bit) and ECDSA (256 bit).
3.3.1 Top 24 Cipher Suites from the Tranco Top 1 Million, as of March 2020:
TLS_AES_256_GCM_SHA384 DHE-RSA-AES256-GCM-SHA384
ECDHE-RSA-AES256-GCM-SHA384 ECDHE-ECDSA-CHACHA20-POLY1305
ECDHE-RSA-AES128-GCM-SHA256 TLS_CHACHA20_POLY1305_SHA256
1 Tranco Top 1 Million analysis performed in March 2020, by Scott Helme (https://scotthelme.co.uk/top-1-million-analysis-march-2020/)

8
TLS_AES_128_GCM_SHA256 ECDHE-ECDSA-AES128-GCM-SHA256
ECDHE-RSA-AES256-SHA384 ECDHE-ECDSA-AES256-GCM-SHA384
ECDHE-RSA-CHACHA20-POLY1305 ECDHE-RSA-AES128-SHA256
ECDHE-RSA-AES256-SHA DHE-RSA-AES128-GCM-SHA256
AES256-GCM-SHA384 DHE-RSA-CHACHA20-POLY1305
AES256-SHA256 ECDHE-ECDSA-AES128-SHA256
AES128-SHA256 DHE-RSA-AES256-SHA256
AES128-GCM-SHA256 ECDHE-ECDSA-AES128-SHA
ECDHE-RSA-AES128-SHA DHE-RSA-AES128-SHA256
3.3.2 Prevention of Weak Ciphers

The firewall offering will be expected to protect against the use of ciphers that are known to offer either weak
protection or none at all, including (but not limited to):
• Null ciphers (no encryption of data provided)

• Anonymous ciphers (no authentication provided)
3.3.3 Support for TLS 1.3
Within the past year, TLS 1.3 was approved by the IETF to be the new standard for securing connections. TLS 1.3
includes several controversial changes over TLS 1.2, along with a few new handshake protocols. Some aspects of TLS
1.3, such as certificate pinning, prevent the third-party inspection of traffic required to protect users from attacks.
As such, secure and desirable behavior is for a firewall offering to force a downgrade from TLS 1.3 to TLS 1.2 and
maintain inspection rather than lose all visibility and protections. The firewall offering will be expected to either
decrypt and inspect TLS 1.3 connections or block TLS 1.3 and force a downgrade to TLS 1.2, and then decrypt and
inspect traffic for functional testing as well as performance testing.
• TLS_AES_256_GCM_SHA384
• ECDHE-RSA-AES256-GCM-SHA384
• ECDHE-RSA-AES128-GCM-SHA256
• TLS_AES_128_GCM_SHA256
• ECDHE-RSA-AES256-SHA384
• ECDHE-RSA-CHACHA20-POLY1305
Results will be reported so that customers will know what behavior(s) to expect from the tested firewall offering.
3.4 Decryption Bypass Exceptions

The firewall offering will be expected to support the configuration of policies that permit conditional bypass of
decryption to preserve privacy, either for regulatory or other reasons. The firewall must maintain decryption
capabilities as tested under section 3.3, concurrently with these conditional bypass rules. Turning off all decryption
on a firewall offering is not an acceptable method for meeting requirements under this section. The firewall offering
will be tested for decryption bypass capabilities under various conditions, including:
• Layer 3 information (i.e., bypass based on source or destination IP address)

• Server Name Indication (SNI) TLS extension information
• Site category based on Common Name (CN) and/or Subject Alternative Name (SAN)

9
3.5 Certificate Validation

The firewall offering will be expected to validate the status of all SSL/TLS certificates presented. When presented
with an invalid certificate, the firewall offering must either prevent the establishment of a connection or replicate
the original invalid status in the proxied/resigned certificate presented to the client, such that the client is aware of
the potential risk.
3.6 TLS Session Re-use

To improve performance and reduce the overhead associated with conducting the full handshake for each session,
the TLS protocol allows for abbreviated handshakes, re-using previously established sessions. The two primary
methods for session re-use are session IDs and session tickets. Whereas session IDs are included in the main TLS
specification, session tickets are an extension of the specification, detailed in a separate RFC. Support for both of
these methods will be tested under this section.

10
4 Threat Prevention
CyberRatings security effectiveness tests verify that an offering can accurately block and log threats while remaining
resistant to false positives. Testing leverages the deep expertise of CyberRatings engineers, utilizing multiple
commercial, open-source and proprietary tools to employ attack methods that are currently being used by
cybercriminals and other threat actors.
Vendors will be provided with a baseline sample set of malicious software ahead of testing to ensure their products
are functioning correctly. These baseline samples will be used to verify basic protection capabilities only at the start
of the test and will not count toward final security effectiveness scores.
The latest signature pack is acquired from the vendor's support site, and the device is deployed using vendor-
provided settings. The signature pack version is recorded for future reference. The vendor may not tune the device.
Once deployed, the device's inspection capabilities are governed solely through firmware and signature updates. All
signatures used must be available to the general public at the time of testing; no custom signatures are permitted.
CyberRatings research has found that this approach reflects a typical deployment and will align results from testing
with product performance in the field. The firewall is required to block and log exploit attempts and malicious
traffic.
4.1 False Positives

False positives are any legitimate, non-malicious content/traffic perceived to be malicious. The ability to correctly
identify and allow legitimate traffic while maintaining protection against malware, exploits and phishing attacks is a
key to effective protection. CyberRatings false positive tests flex the ability of the firewall to block attacks while
permitting legitimate traffic. Security products that block legitimate traffic will have the sensitivity of those
protections turned down or disabled to allow legitimate traffic.
If a device experiences false positive events, it will be tuned until no further false positive events are encountered.
4.1.1 Initial check – legitimate traffic, documents, and files
This test transmits a varied sample of legitimate application traffic, documents, and files that should be identified
and allowed or blocked based on policy rules. Testing may include but is not limited to the following file formats:
HTML, .js, .exe, .jar, .xlsm, .css, .pdf, .ppt, .pptx, .doc, .docx, .zip, .DLL, .xls, .xlsx, .chm, .rar, .Ink, .cur, .tar, .xrc.
4.1.2 Ongoing check – legitimate traffic, documents, and files
Since firewalls may include a cloud offering which utilizes machine learning to modify/tune settings in real-time,
testing for false positives requires legitimate traffic and documents to be included when testing the firewall’s ability
to block attacks. CyberRatings will introduce legitimate traffic, documents, and files into tests in sections 4.2, 4.3,
and 4.4, including but not limited to the following file formats: HTML, .js, .exe, .jar, .xlsm, .css, .pdf, .ppt, .pptx, .doc,
.docx, .zip, .DLL, .xls, .xlsx, .chm, .rar, .Ink, .cur, .tar, .xrc.
4.2 Exploits
While vulnerabilities are patched, and defenses against exploits are incorporated into new versions of operating
systems such as Windows, many organizations cannot easily upgrade due to financial, technical, or other
constraints. And often, the most valuable assets have the most stringent change control to avoid business
interruption. This creates a challenging dynamic whereby the most valuable assets tend to be the most difficult to
defend (e.g., older OS, unpatched, etc.). Therefore, as vulnerabilities are patched and defenses against exploits are
incorporated into new versions of operating systems, the value of a firewall is often associated with its ability to
protect older, unpatched, and generally more vulnerable systems.
CyberRatings security effectiveness testing leverages our engineers' deep expertise, who utilize multiple
commercial, open-source, and proprietary tools as appropriate. With thousands of exploits, this is the industry's

11
most comprehensive test to date. Most notably, all of the live exploits and payloads in the CyberRatings exploit test
have been validated in our lab such that one or more of the following are true:
• A reverse shell is returned

• A bind shell is opened on the target allowing the attacker to execute arbitrary commands
• Arbitrary code is executed
• A malicious payload is installed
• A system is rendered unresponsive
• Etc.
This test attempts to exploit specific vulnerabilities on victim systems within a protected environment. Various
vulnerable applications and system features are used, and multiple payloads are employed with them.
4.3 Protection Resiliency

Different variations of an exploit can be used to exploit a vulnerability. And many security vendors claim their
products provide vulnerability-based protection that will block exploitation of vulnerabilities regardless of the
specific exploit. A product that can defend against multiple exploit variations provides resilient protection.
Below is an example list of evasions that may be used. It is not a comprehensive list but is intended to illustrate the
kinds of evasions CyberRatings will employ during testing. Vendors should protect against these evasions and others
like them. Providing protection for only these evasions but not others ("studying to pass the test") will likely fail
CyberRatings evasion testing.
• res-esc-001 Hex encoded VBScript decoded using JavaScript unescape
• res-esc-002 Hex encoded VBScript as variable decoded using JavaScript unescape
• res-sep-001 External VBScript file loaded from HTML
• res-sep-002 Multiple VBScript files loaded from HTML
• res-sep-003 Multiple VBScript files loaded with external JavaScript file
• res-nb-001 VBScript interspersed randomly with null bytes
• res-pay-001 nishang bind shell obfuscated with Unicorn
• res-pay-002 native Unicorn generated bind shell
• res-pay-003 nishang bind shell obfuscated with PowerSploit's Out-EncodedCommand
• res-pay-004 Veil Ordnance bind shell shellcode dropped into PowerSploit's Invoke-Shellcode; then obfuscated with PowerSploit's Out-
EncodedCommand
• res-pay-005 custom bind shell shellcode obfuscated with Invoke-Obfuscation
• res-pay-006 custom bind shell shellcode with password prompt obfuscated with Invoke-Obfuscation
• res-mth-mrg-ord-pay-spl-chr-wsp-001 numeric values/equations modified and/or inserted; hexadecimal values replaced with decimal
values; combine 'myarray' instantiation into a single line; combine powershell command into single line; Remove runmumaa and add to
setnotsafemode function; move setnotsafemode function to bottom of script; Some strings split with "+" and "&"; some lines split with "_";
some script commands/strings converted to series of chr()/Clng/&H; both spaces and linefeeds replaced with multiples of each; nishang
bind shell obfuscated with Unicorn
• res-mth-mrg-ord-pay-spl-chr-ws-pch-001 numeric values/equations modified and/or inserted; hexadecimal values replaced with decimal
values; combine 'myarray' instantiation into single line; combine powershell command into single line; Remove runmumaa and add to
bind shell obfuscated with Unicorn; chunked
some script commands/strings converted to series of chr()/Clng/&H; both spaces and linefeeds replaced with multiples of each; native
Unicorn generated bind shell
• res-mth-mrg-ord-pay-spl-chr-wsp-cg-002 numeric values/equations modified and/or inserted; hexadecimal values replaced with decimal
some script commands/strings converted to series of chr()/Clng/&H; both spaces and linefeeds replaced with multiples of each; native
Unicorn generated bind shell; chunked and gzip compressed

12
bind shell obfuscated with PowerSploit's Out-EncodedCommand
• res-mth-mrg-ord-pay-spl-chr-wsp-cd-003 numeric values/equations modified and/or inserted; hexadecimal values replaced with decimal
some script commands/strings converted to series of chr()/Clng/&H using online vbscript obfuscator; both spaces and linefeeds replaced
with multiples of each; nishang bind shell obfuscated with PowerSploit's Out-EncodedCommand; chunked and deflate compressed
some script commands/strings converted to series of chr()/Clng/&H; both spaces and linefeeds replaced with multiples of each; Veil
Ordnance bind shell shellcode dropped into PowerSploit's Invoke-Shellcode; then obfuscated with PowerSploit's Out-EncodedCommand
• res-mth-mrg-ord-pay-spl-chr-wsp-ch-004 numeric values/equations modified and/or inserted; hexadecimal values replaced with decimal
some script commands/strings converted to series of chr()/Clng/&H; both spaces and linefeeds replaced with multiples of each; Veil
Ordnance bind shell shellcode dropped into PowerSploit's Invoke-Shellcode; then obfuscated with PowerSploit's Out-EncodedCommand;
chunked
some script commands/strings converted to series of chr()/Clng/&H; both spaces and linefeeds replaced with multiples of each; custom
bind shell shellcode obfuscated with Invoke-Obfuscation
• res-mth-mrg-ord-pay-spl-chr-wsp-cg-005 numeric values/equations modified and/or inserted; hexadecimal values replaced with decimal
bind shell shellcode obfuscated with Invoke-Obfuscation; chunked and gzip compressed
• res-mth-mrg-ord-pay-splc-hrw-sp-006 numeric values/equations modified and/or inserted; hexadecimal values replaced with decimal
bind shell shellcode with password prompt obfuscated with Invoke-Obfuscation
• res-mth-mrg-ord-pay-spl-chr-wsp-cd-006 numeric values/equations modified and/or inserted; hexadecimal values replaced with decimal
bind shell shellcode with password prompt obfuscated with Invoke-Obfuscation; chunked and deflate compressed
• res-ren-chr-wsp-pay-mth-spl-001 procedures and variables renamed; some script commands/strings converted to series of chr()/Clng/&H;
both spaces and linefeeds replaced with multiples of each; numeric values/equations modified and/or inserted; hexadecimal values
replaced with decimal values; Some strings split with "+" and "&"; some lines split with "_"; nishang bind shell obfuscated with Unicorn
• res-ren-chr-wsp-pay-mth-spl-ch-001 procedures and variables renamed; some script commands/strings converted to series of
chr()/Clng/&H; both spaces and linefeeds replaced with multiples of each; numeric values/equations modified and/or inserted; hexadecimal
values replaced with decimal values; Some strings split with "+" and "&"; some lines split with "_"; nishang bind shell obfuscated with
Unicorn; chunked
replaced with decimal values; Some strings split with "+" and "&"; some lines split with "_"; native Unicorn generated bind shell
• res-ren-chr-wsp-pay-mth-spl-cg-002 procedures and variables renamed; some script commands/strings converted to series of
values replaced with decimal values; Some strings split with "+" and "&"; some lines split with "_"; native Unicorn generated bind shell;
chunked and gzip compressed
replaced with decimal values; Some strings split with "+" and "&"; some lines split with "_"; nishang bind shell obfuscated with PowerSploit's
Out-EncodedCommand
• res-ren-chr-wsp-pay-mth-spl-cd-003 procedures and variables renamed; some script commands/strings converted to series of
values replaced with decimal values; Some strings split with "+" and "&"; some lines split with "_"; nishang bind shell obfuscated with
PowerSploit's Out-EncodedCommand; chunked and deflate compressed

13
replaced with decimal values; Some strings split with "+" and "&"; some lines split with "_"; Veil Ordnance bind shell shellcode dropped into
PowerSploit's Invoke-Shellcode; then obfuscated with PowerSploit's Out-EncodedCommand
• res-ren-chr-wsp-pay-mth-spl-ch-004 procedures and variables renamed; some script commands/strings converted to series of
values replaced with decimal values; Some strings split with "+" and "&"; some lines split with "_"; Veil Ordnance bind shell shellcode
dropped into PowerSploit's Invoke-Shellcode; then obfuscated with PowerSploit's Out-EncodedCommand; chunked
replaced with decimal values; Some strings split with "+" and "&"; some lines split with "_"; custom bind shell shellcode obfuscated with
Invoke-Obfuscation
• res-ren-chr-wsp-pay-mth-spl-cg-005 procedures and variables renamed; some script commands/strings converted to series of
values replaced with decimal values; Some strings split with "+" and "&"; some lines split with "_"; custom bind shell shellcode obfuscated
with Invoke-Obfuscation; chunked and gzip compressed
replaced with decimal values; Some strings split with "+" and "&"; some lines split with "_"; custom bind shell shellcode with password
prompt obfuscated with Invoke-Obfuscation
• res-ren-chr-wsp-pay-mth-spl-cd-006 procedures and variables renamed; some script commands/strings converted to series of
values replaced with decimal values; Some strings split with "+" and "&"; some lines split with "_"; custom bind shell shellcode with
password prompt obfuscated with Invoke-Obfuscation; chunked and deflate compressed
• res-wsp-001 both spaces and linefeeds replaced with multiples of each
• res-ren-001 procedures and variables renamed
• res-mth-001 numeric values/equations modified and/or inserted; hexadecimal values replaced with decimal values
• res-chr-001 change all chr() to chrw() and vice versa where possible
• res-chr-002 change chr() and chrw() to chrb()
• res-chr-003 some script commands/strings converted to series of chr()/Clng/&H using online vbscript obfuscator
• res-pay-007 Veil Ordnance bind shell shellcode dropped into PowerSploit's Invoke-Shellcode; then obfuscated with PowerSploit's Out-
EncodedCommand
• res-pay-008 Use wscript to call original payload (PoshRat method)
• res-pay-009 nishang bind shell obfuscated with Unicorn
• res-ord-001 Remove runmumaa and add to setnotsafemode function; move setnotsafemode function to bottom of script
• res-spl-001 Some strings split with "+" and "&"; some lines split with "_"
• res-mrg-001
• combine 'myarray' instantiation into single line; combine powershell command into single line
• res-ren-chr-001 Combination of techniques used in res-ren-001 and res-chr-003
• res-ren-chr-wsp-001 Combination of techniques used in res-ren-001; res-chr-003; and res-wsp-001
• res-ren-chr-wsp-pay-001 Combination of techniques used in res-ren-001; res-chr-003; res-wsp-001; and res-pay-004
• res-ren-pay-001 Combination of techniques used in res-ren-001 and res-pay-007
• res-ren-chr-wsp-pay-mth-001 Combination of techniques used in res-ren-001; res-chr-003; res-wsp-001; res-pay-007; and res-mth-001
• res-mth-mrg-001 Combination of techniques used in res-mth-001 and res-mrg-001
• res-mth-mrg-ord-001 Combination of techniques used in res-mth-001; res-mrg-001; and res-ord-001
• res-mth-mrg-ord-pay-001 Combination of techniques used in res-mth-001; res-mrg-001; res-ord-001; and res-pay-008
• res-mth-mrg-ord-pay-spl-001 Combination of techniques used in res-mth-001; res-mrg-001; res-ord-001; res-pay-008; and res-spl-001
• res-mth-mrg-ord-pay-spl-chr-001 Combination of techniques used in res-mth-001; res-mrg-001; res-ord-001; res-pay-008; res-spl-001; and
res-chr-003
• res-mth-mrg-ord-pay-spl-chr-002 Combination of techniques used in res-mth-001; res-mrg-001; res-ord-001; res-pay-008; res-spl-001; and
res-chr-003; plus removal of all CLng's
• res-mth-mrg-ord-pay-chr-001 Combination of techniques used in res-mth-001; res-mrg-001; res-ord-001; res-pay-008; and res-chr-003
• res-mth-mrg-ord-pay-spl-chr-wsp-007 Combination of techniques used in res-mth-001; res-mrg-001; res-ord-001; res-pay-008; res-spl-001;
res-chr-003; res-wsp-001; plus removal of all CLng's
• res-mth-mrg-ord-pay-spl-chr-wsp-008 Combination of techniques used in res-mth-001; res-mrg-001; res-ord-001; res-pay-009; res-spl-001;
res-chr-003; res-wsp-001; res-ren-001; plus removal of all CLng's; replace 'LANGUAGE="VBScript"' with 'type="text/vbScript"'
• combo-001 UTF-8 encoding; HTTP/1.1 chunked response with chunk sizes preceded by multiple zeros (hex '30'); small TCP segments; small
IP fragments; padding
• combo-002 UTF-8 encoding with BOM; HTTP/1.1 chunked response with chunk sizes followed by backspace (hex '08'); small TCP segments;
small IP fragments in reverse order; padding

14
• combo-003 UTF-16 encoding with BOM; HTTP/1.1 chunked response with chunk sizes followed by end of text (hex '03'); small TCP
segments in random order; small IP fragments; padding
• combo-004 UTF-8 encoding; no http or html declarations; HTTP/1.1 chunked response with chunk sizes followed by escape (hex '1b'); small
TCP segments; small IP fragments in random order; padding
• combo-005 UTF-8 encoding with BOM; no http or html declarations; HTTP/1.1 chunked response with chunk sizes followed by null (hex
'00'); small TCP segments in random order; small IP fragments in reverse order; padding
4.4 Evasions
Threat actors deploy evasions to disguise and modify attacks at the point of delivery to avoid detection by security
products. Therefore, it is imperative that a firewall offering correctly handles evasions.
Attackers can modify attacks and malicious code in a number of ways order to evade detection. If an firewall fails to
detect a single form of evasion, a attack can bypass protection, rendering it ineffective. CyberRatings verifies that
the firewall is capable of detecting and blocking exploits and malware when subjected to varying common evasion
techniques. Wherever possible, the firewall is expected to successfully decode the obfuscated traffic to provide an
accurate alert relating to the original attack, rather than alerting purely on anomalous traffic detected as a result of
the evasion technique itself.
A number of common attacks are executed to ensure that they are detected in their unmodified state. These will be
chosen from a suite of older/common basic attacks for which CyberRatings is certain that all vendors will have
protection.
4.4.1 IP Packet Fragmentation
These tests determine the effectiveness of the fragment reassembly mechanism of the firewall.
• Fragments from 8 to 32 bytes in size

• Ordered, out-of-order, or reverse order fragments
• Fragment overlap, favoring new and favoring old data
• Interleaved, duplicate, duplicate with or without incrementing DWORD, duplicate packets with random
payload, or duplicate packets scheduled for later delivery
• Any combination of the above methods
Some Examples:
o Ordered 8 byte fragments
o Ordered 24 byte fragments
o Out of order 8 byte fragments
o Ordered 8 byte fragments, duplicate last packet
o Out of order 8 byte fragments, duplicate last packet
o Ordered 8 byte fragments, reorder fragments in reverse
o Ordered 16 byte fragments, fragment overlap (favor new)
o Ordered 16 byte fragments, fragment overlap (favor old)
o Out of order 8 byte fragments, interleaved duplicate packets scheduled for later delivery
It is a requirement of the test that the firewall submitted should have all IP fragmentation reassembly options
enabled by default in the shipping product.
4.4.2 TCP Stream Segmentation
These tests determine the effectiveness of the stream reassembly mechanism of the firewall.
• Segments from 1 - 2048 bytes in size

• Ordered, reverse ordered, or out-of-order segments, with favor old or favor new
• Duplicate, duplicate interleaved, duplicate last packet, or overlapping segments
• Invalid or NULL TCP control flags

15
• Sequence resync requests, random initial sequence number, or out-of-window sequence numbers
• Faked retransmits, protection against wrapping sequence (PAWS) numbers, or segments containing
random data
• Endianness interchanged
• Any combination of the above methods
Some examples:
o Ordered 1 byte segments, interleaved duplicate segments with invalid TCP checksums
o Ordered 1 byte segments, interleaved duplicate segments with null TCP control flags
o Ordered 1 byte segments, interleaved duplicate segments with requests to resync sequence numbers
mid-stream
o Ordered 1 byte segments, duplicate last packet
o Ordered 2 byte segments, segment overlap (favor new)
o Ordered 1 byte segments, interleaved duplicate segments with out-of-window sequence numbers
o Out of order 1 byte segments
o Out of order 1 byte segments, interleaved duplicate segments with faked retransmits
o Ordered 1 byte segments, segment overlap (favor new)
o Out of order 1 byte segments, PAWS elimination (interleaved duplicate segments with older TCP
timestamp options)
o Ordered 16 byte segments, segment overlap (favor new (Unix))
It is a requirement of the test that the firewall submitted should have all TCP stream reassembly options enabled by
default in the shipping product.
4.4.3 HTTP Obfuscation
Web browsers request content from servers over HTTP using the ASCII character-set. HTTP encoding replaces
unsafe non-ASCII characters with a "%" followed by two hexadecimal digits. Web servers and clients understand
how to decode the request and responses. However, this mechanism can be abused to circumvent protection that is
looking to match specific strings of characters.
Chunked encoding allows the server to break a document into smaller chunks and transmit them individually. The
server needs only to specify the size of each chunk before it is transmitted and then indicate when the last chunk
has been transmitted. Since chunked encoding intersperses arbitrary numbers (chunk sizes) with the elements of
the original document, it can be used to greatly change the appearance of the content as observed "on the wire"
during transmission. In addition, the server can choose to break the document into chunks at arbitrary points. This
makes it difficult to reliably identify the original HTML content from the raw data on the network.
Below is an example list of evasions that may be used. It is not a comprehensive list, but is intended to illustrate the
kinds of evasions CyberRatings will employ during testing. Vendors should provide protection against these evasions
and others like them. Providing protection for only these evasions but not others ("studying to pass the test") will
likely result in failure of CyberRatings evasion testing.
• Declared HTTP/0.9 response; but includes response headers; chunking declared but served without chunking
• HTTP/1.1 chunked response with chunk sizes preceded by multiple zeros (hex '30')
• HTTP/1.1 chunked response with chunk sizes followed by backspace (hex '08')
• HTTP/1.1 chunked response with chunk sizes followed by end of text (hex '03')
• HTTP/1.1 chunked response with chunk sizes followed by escape (hex' 1b')
• HTTP/1.1 chunked response with chunk sizes followed by null (hex '00')
• HTTP/1.1 chunked response with chunk sizes followed by a space (hex '20') then a zero (hex '30')
• HTTP/1.1 chunked response with final chunk size of
'00000000000000000000000000000000000000000000000000000000000000000000000000000000' (rather than '0')
• HTTP/1.1 response with line folded transfer-encoding header declaring chunking ('Transfer-Encoding: ' followed by CRLF (hex '0d 0a')
followed by 'chunked' followed by CRLF (hex '0d 0a'); served without chunking

16
• HTTP/1.1 response with transfer-encoding header declaring chunking with lots of whitespace ('Transfer-Encoding:' followed by 8000 spaces
(hex '20' * 8000) followed by 'chunked' followed by CRLF (hex '0d 0a'); served chunked
• HTTP/1.0 response declaring chunking; served without chunking
• HTTP/1.0 response declaring chunking with invalid content-length header; served without chunking
• HTTP/1.1 response with "\tTransfer-Encoding: chunked"; served chunked
• HTTP/1.1 response with "\tTransfer-Encoding: chonked" after custom header line with "chunked" as value; served without chunking
• HTTP/1.1 response with header with no field name and colon+junk string; followed by '\tTransfer-Encoding: chunked' header; followed by
custom header; served chunked
• HTTP/1.1 response with "\r\rTransfer-Encoding: chunked"; served chunked
• HTTP/1.1 response with using single "\n"'s instead of "\r\n"'s; chunked
• HTTP/1.1 response with \r\n\r\n before first header; chunked
• HTTP/1.1 response with "SIP/2.0 200 OK\r\n" before status header; chunked
• HTTP/1.1 response with space+junk string followed by \r\n before first header; chunked
• HTTP/1.1 response with junk string before status header; chunked
• HTTP/1.1 response with header end \n\014\n\n; chunked
• HTTP/1.1 response with header end \r\n\016\r\n\r\n; chunked
• HTTP/1.1 response with header end \n\r\r\n; chunked
• HTTP/1.1 response with header end \n\017\018\n\n; chunked
• HTTP/1.1 response with header end \n\030\n\019\n\n; chunked
• HTTP/1.1 response with status code -203.030; with message-body; chunked
• HTTP/1.1 response with status code 402; with message-body; chunked
• HTTP/1.1 chunked response with no status indicated
• No status line; chunking indicated; served unchunked
• HTTP/1.1 response with invalid content-length header size declaration followed by space and null (hex '20 00')
• HTTP/1.01 declared; served chunked
• HTTP/01.1 declared; served chunked
• HTTP/2.B declared; served chunked
• HTTP/9.-1 declared; served chunked
• Double Transfer-Encoding: first empty; last chunked. Served with invalid content-length; not chunked.
• Relevant headers padded by preceding with hundreds of random custom headers
4.4.4 HTTP Compression

Per RFC 2616, the HTTP protocol allows the client to request and the server to use several compression methods.
These compression methods not only improve performance in many circumstances, they completely change the
characteristic size and appearance of HTML documents.
like them. Providing protection for only these evasions but not others ("studying to pass the test") will likely result in
CyberRatings evasion testing failure.
• HTTP/1.1 response compressed with gzip; invalid content-length
• HTTP/1.1 response declaring gzip followed by junk string; invalid content-length; served uncompressed
• HTTP/1.1 response compressed with deflate; invalid content-length
• HTTP/1.1 response declaring deflate followed by junk string; invalid content-length; served uncompressed
• HTTP/1.1 response with content-encoding declaration of gzip followed by space+junk string; served uncompressed and chunked
• HTTP/1.1 response with content-encoding header for deflate; followed by content-encoding header for gzip; served uncompressed and
chunked
• HTTP/1.1 chunked response with chunk sizes preceded by multiple zeros (hex '30'); compressed with gzip
• HTTP/1.1 chunked response with chunk sizes followed by backspace (hex '08'); compressed with gzip
• HTTP/1.1 chunked response with chunk sizes followed by end of text (hex '03'); compressed with gzip

17
• HTTP/1.1 chunked response with chunk sizes followed by escape (hex' 1b'); compressed with gzip
• HTTP/1.1 chunked response with chunk sizes followed by null (hex '00'); compressed with gzip
• HTTP/1.1 chunked response with chunk sizes followed by a space (hex '20') then a zero (hex '30'); compressed with gzip
• HTTP/1.1 chunked response with chunk sizes preceded by multiple zeros (hex '30'); compressed with deflate
• HTTP/1.1 chunked response with chunk sizes followed by backspace (hex '08'); compressed with deflate
• HTTP/1.1 chunked response with chunk sizes followed by end of text (hex '03'); compressed with deflate
• HTTP/1.1 chunked response with chunk sizes followed by escape (hex' 1b'); compressed with deflate
• HTTP/1.1 chunked response with chunk sizes followed by null (hex '00'); compressed with deflate
• HTTP/1.1 chunked response with chunk sizes followed by a space (hex '20') then a zero (hex '30'); compressed with deflate
4.4.5 HTML Obfuscation

HTML is a filetype that web servers transmit via HTTP to web browsers. Whereas HTTP obfuscations evade detection
by misusing the transmission, HTML obfuscations are contained within the content itself.
like them. Providing protection for only these evasions but not others ("studying to pass the test") will likely result in
CyberRatings evasion testing failure.
• js-binary-obfuscation • UTF-16-BE encoding without BOM
• babel-minify • UTF-16-LE encoding without BOM; no http or html declarations
• closure • UTF-16-BE encoding without BOM; no http or html declarations
• code-protect • UTF-7 encoding
• confusion • UTF-8 encoding
• jfogs • UTF-8 encoding
• jfogs-reverse • EICAR string included at top of HTML
• jjencode • Hex encoded script decoded using JavaScript unescape
• jsbeautifier • Unicode encoded script decoded using JavaScript unescape
• jsmin • Hex encoded script as variable decoded using JavaScript unescape
• js-obfuscator • Unicode encoded script as variable decoded using JavaScript unescape
• qzx-obfuscator • padded with <=5MB
• chunked and gzip compressed js-binary-obfuscation • padded with <=25MB
• chunked and deflate compressed js-binary-obfuscation • padded with >25MB
• UTF-8 encoding • padded with <=5MB; chunked and compressed with gzip
• UTF-8 encoding with BOM • padded with <=25MB; chunked and compressed with gzip
• UTF-16 encoding with BOM • padded with >25MB; chunked and compressed with gzip
• UTF-8 encoding; no http or html declarations • padded with <=5MB; chunked and compressed with deflate
• UTF-8 encoding with BOM; no http or html declarations • padded with <=25MB; chunked and compressed with deflate
• UTF-16 encoding with BOM; no http or html declarations • padded with >25MB; chunked and compressed with deflate
• UTF-16-LE encoding without BOM

18
5 Performance
This section measures the performance of a device using various traffic conditions that provide metrics for real-
world performance. Individual implementations will vary based on usage; however, these quantitative metrics
provide a gauge as to whether a particular device is appropriate for a given environment.
Test cases may be configured in both uni-directional or bi-directional mode as needed to represent enterprise use
cases for various applications.
5.1 Raw Packet Processing Performance (UDP Throughput)

This test uses UDP packets of varying sizes generated by traffic generation appliances. A constant stream of the
appropriate packet size—with variable source and destination IP addresses transmitting from a fixed source port to
a fixed destination port—is transmitted bi-directionally through each port pair of the device. Each packet contains
dummy data and is targeted at a valid port on a valid IP address on the target subnet. Network monitoring tools
verify the percentage load and frames per second (fps) figures across each inline port pair before each test begins.
Multiple tests are run, and averages are taken where necessary.
This traffic does not attempt to simulate any form of real-world network condition. No TCP sessions are created
during this test, and the detection engine has very little to do. However, each vendor will be required to write a
signature to detect the test packets to ensure that they are being passed through the detection engine and not
"fast-tracked" from the inbound port to the outbound port.
This test aims to determine the raw packet processing capability of each inline port pair of the device as well as its
effectiveness at forwarding packets quickly to provide the highest level of network performance with the lowest
latency.
5.1.1 64 Byte Packets
Maximum 1,488,000 frames per second per Gigabit of traffic. This test determines a device's ability to process
packets from the wire under the most challenging packet processing conditions.
Maximum 844,000 frames per second per Gigabit of traffic
Maximum 452,000 frames per second per Gigabit of traffic.
Maximum 234,000 frames per second per Gigabit of traffic. This test provides a reasonable indication of a device's
ability to process packets from the wire on an "average" network.
Maximum 81,000 frames per second per Gigabit of traffic. This test has been included to demonstrate how easy it is
to achieve good results using large packets. Readers should use caution when considering those test results that
quote only performance figures using similar packet sizes.
19
5.2 Latency
The latency and user response time test goal is to determine the effect the device has on traffic passing through it
under various load conditions. Test traffic is passed across the infrastructure switches and through all inline port
pairs of the device simultaneously (the latency of the basic infrastructure is known and is constant throughout the
tests).
Packet loss and average latency (µs) are recorded for each packet size (64, 128, 256, 512, 1,024, and 1,514 bytes) at
a load level of 90% of the maximum throughput with zero packet loss, as previously determined in section 5.1.
5.2.1 64 Byte Frames
Maximum 1,488,000 frames per second per Gigabit of traffic
5.2.2 128 Byte Frames
5.2.5 1,024 Byte Packets
5.2.6 1,514 Byte Packets
5.3 Maximum Capacity

The use of traffic generation appliances allows CyberRatings engineers to create true "real-world" traffic at multi-
Gigabit speeds as a background load for the tests.
The goal is to stress the inspection engine and determine how it handles high volumes of TCP connections per
second, application layer transactions per second, and concurrent open connections. All packets contain valid
payload and address data, and these tests provide an excellent representation of a live network at various
connection/transaction rates.
Note that in all tests, the following critical "breaking points" – where the final measurements are taken – are used:
• Excessive concurrent TCP connections – Latency within the firewall is causing an unacceptable increase in open
connections.
• Excessive concurrent HTTP connections – Latency within the firewall is causing excessive delays and increased
response time.
• Unsuccessful HTTP transactions – Normally, there should be zero unsuccessful transactions. Once these appear,
it is an indication that excessive latency within the firewall is causing connections to time out.
5.3.1 Theoretical Maximum Concurrent TCP Connections
This test is designed to determine the device's maximum concurrent TCP connections with no data passing across
the connections. This type of traffic would not typically be found on a normal network, but it provides the means to
determine the maximum possible concurrent connections.
20
An increasing number of Layer 4 TCP sessions are opened through the device. Each session is opened normally and
then held open for the test's duration as additional sessions are added up to the maximum possible. The load is
increased until no more connections can be established, and this number is recorded.
5.3.2 Maximum TCP Connections per Second
This test is designed to determine the maximum TCP connection rate of the device with one byte of data passing
across the connections. This type of traffic would not typically be found on a normal network, but it provides the
means to determine the maximum possible TCP connection rate.
An increasing number of new sessions are established through the device and ramped slowly to determine the exact
point of failure. Each session is opened normally, one byte of data is passed to the host, and then the session is
closed immediately. The load is increased until one or more of the breaking points defined earlier is reached.
5.3.3 Maximum HTTP Connections per Second
This test is designed to determine the maximum TCP connection rate of the device with a 1-byte HTTP response
size. The response size defines the number of bytes contained in the body, excluding any bytes associated with the
HTTP header. A 1-byte response size is designed to provide theoretical maximum HTTP connections per second rate.
Client and server are using HTTP 1.0 without keep-alive, and the client will open a TCP connection, send one HTTP
request, and close the connection. This ensures that all TCP connections are closed immediately upon the request
being satisfied; thus any concurrent TCP connections will be caused purely as a result of latency the device
introduces on the network. The load is increased until one or more of the breaking points defined earlier is reached.
5.3.4 Maximum HTTP Transactions per Second
This test is designed to determine the maximum HTTP transaction rate of the device with a 1-byte HTTP response
size. The object size defines the number of bytes contained in the body, excluding any bytes associated with the
HTTP header. A 1-byte response size is designed to provide a theoretical maximum connections per second rate.
Client and server are using HTTP 1.1 with persistence, and the client will open a TCP connection, send 10 HTTP
requests, and close the connection. This ensures that TCP connections remain open until all 10 HTTP transactions
are complete, thus eliminating the maximum connection per second rate as a bottleneck (one TCP connection = 10
HTTP transactions). The load is increased until one or more of the breaking points defined earlier is reached.
5.3.5 Maximum HTTP(S) Connections per Second
This test is designed to determine the maximum TCP connection rate of the device with a 1-byte HTTP response
size. This type of traffic would not typically be found on a normal network, but it provides the means to measure the
device's maximum possible SSL/TLS handshake rate.
An increasing number of new sessions are established through the device and ramped slowly to determine the exact
point of failure. Each session is opened normally, one byte of data is passed to the host, and then the session is
closed immediately. The load is increased until one or more of the breaking points defined earlier is reached
5.4 HTTP Capacity

These tests aim to stress the HTTP detection engine and determine how the device copes with network loads of
varying average packet size and varying connections per second. By creating genuine session-based traffic with
varying session lengths, the device is forced to track valid TCP sessions, thus ensuring a higher workload than simple
packet-based background traffic. This provides a test environment that is as close to real-world conditions as
possible to achieve in a lab environment while ensuring absolute accuracy and repeatability.
Each transaction consists of a single HTTP GET request, and there are no transaction delays (i.e., the web server
responds immediately to all requests). All packets contain a valid payload (a mix of binary and ASCII objects) and
21
address data. This test provides an excellent representation of a live network (albeit one biased towards HTTP
traffic) at various network loads.
1,000 1,000 1,000 1,000 1,000 45,000

1,000 40,000
35,000
Connections / Second
Megabits per Second
800
30,000
600 25,000
20,000
400 15,000
10,000
200
5,000
0 0
44 KB 21 KB 10 KB 4.5 KB 1.7 KB
Response Response Response Response Response
CPS 2,500 5,000 10,000 20,000 40,000
Mbps 1,000 1,000 1,000 1,000 1,000
5.4.1 44 KB HTTP Response Size – 2,500 Connections per Second

Maximum 2,500 new connections per second per Gigabit of traffic with a 44 KB HTTP response size—maximum
140,000 packets per second per Gigabit of traffic. With relatively low connection rates and large packet sizes, all
hosts should be capable of performing well throughout this test.
185,000 packets per second per Gigabit of traffic. With average connection rates and average packet sizes, this is a
good approximation of a real-world production network. All hosts should be capable of performing well throughout
this test.
225,000 packets per second per Gigabit of traffic. With smaller packet sizes coupled with high connection rates, this
represents a very heavily used production network.
5.4.4 4.5 KB HTTP Response Size – 20,000 Connections per Second
Maximum 20,000 new connections per second per Gigabit of traffic with a 4.5 KB HTTP response size—maximum
300,000 packets per second per Gigabit of traffic. With small packet sizes and extremely high connection rates, this
is an extreme test for any host.
5.4.5 1.7 KB HTTP Response Size – 40,000 Connections per Second
445,000 packets per second per Gigabit of traffic. With small packet sizes and extremely high connection rates, this
is an extreme test for any host.
5.5 Application Average Response Time: HTTP

Test traffic is passed across the infrastructure switches and through all inline port pairs of the device simultaneously
(the latency of the basic infrastructure is known and is constant throughout the tests). The results are recorded at
22
each response size (44 KB, 21 KB, 10 KB, 4.5 KB, and 1.7 KB HTTP responses) at a load level of 90% of the maximum
throughput with zero packet loss as previously determined in section 5.4.
5.6 HTTPS Capacity

These tests aim to stress the HTTPS engine and determine how the device copes with network loads of varying
average packet size and varying connections per second. By creating session-based traffic with varying session
lengths, the device is forced to track valid TCP sessions, thus ensuring a higher workload than simple packet-based
background traffic. This provides a test environment that is as close to real-world conditions as possible to achieve
in a lab environment while ensuring accuracy and repeatability.
Each transaction consists of a single (1) HTTP(S) GET request, and there are no transaction delays (i.e., the
webserver responds immediately to all requests). All packets contain a valid payload (a mix of binary and ASCII
objects) and address data. This test provides a feasible representation of a live network (albeit one biased towards
HTTPS traffic) at various network loads.
The same cipher selection methodology outlined in Section 3.2 will be used to determine testing targets under this
section. The top four ciphers as listed in Section 3.3.1:
• TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 using 2,048 bit and 4,096 bit keys

• TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256 using a 2,048 bit key
• TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256 using a 256 bit key and the secp256r1 curve
• TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384 using a 2,048 bit key
• TLS13_AES256_GCM_SHA384
5.6.1 1.7 KB HTTPS Response Size – 40,000 Connections per Second
445,000 packets per second per Gigabit of traffic.
5.6.2 4.5 KB HTTPS Response Size – 20,000 Connections per Second
5.6.3 10 KB HTTPS Response Size – 10,000 Connections per Second
5.7 Application Average Response Time: HTTPS

Test traffic is passed across the infrastructure switches and through all inline port pairs of the device simultaneously
(the latency of the basic infrastructure is known and is constant throughout the tests). The results are recorded at
23
each response size (44 KB, 21 KB, 10 KB, 4.5 KB, and 1.7 KB HTTP responses) at a load level of 90% of the maximum
throughput with zero packet loss previously determined in section 5.6.
5.8 "Real-World" Single Application Flows

Where previous tests provide a pure HTTP environment with varying connection rates and average packet sizes, this
test aims to simulate real-world single application traffic.
• Single Application SIP Flow • Single Application RDP Flow

• Single Application FIX Flow • Single Application YouTube Flow
• Single Application SMTP Flow • Single Application WebEx Flow
• Single Application FTP Flow • Single Application MSSQL Flow
• Single Application SMB Flow
24
6 Stability and Reliability

Long-term stability is particularly important for an inline device, where failure can produce network outages.
Stability and reliability tests verify the device's stability along with its ability to maintain security effectiveness while
under normal load and while passing malicious traffic. Products that cannot sustain legitimate traffic (or that crash)
while under hostile attack will not pass.
The device must remain operational and stable throughout all tests and block 100% of previously blocked traffic,
raising an alert for each. If any prohibited traffic passes successfully, caused by either the volume of traffic or by the
device failing open for any reason, this will result in a FAIL.
If the device becomes unresponsive during any part of this test and cannot autonomously recover, CyberRatings will
reset or reboot it. Since a device that becomes unresponsive during normal operation would be an operational task
requiring attention, these activities will be reported as such.
6.1 Blocking Under Extended Attack

The device is exposed to a constant stream of security policy violations over an extended period of time. The device
is configured to block and alert, and thus this test provides an indication of the effectiveness of both the blocking
and alert handling mechanisms.
A continuous stream of security policy violations mixed with legitimate traffic is transmitted through the device for
eight hours at a maximum of 100 Mbps, with no additional background traffic. This is not intended as a stress test in
terms of traffic load (covered in the previous section); it is merely a reliability test in terms of consistency of blocking
performance.
The device is expected to remain operational and stable throughout this test and to block 100% of recognizable
violations, raising an alert for each. If any recognizable policy violations are passed, caused by either the volume of
traffic or the device failing open for any reason, this will result in a FAIL.
6.2 Passing Legitimate Traffic under Extended Attack

This test is identical to test 6.1 where the external interface of the device is exposed to a constant stream of exploits
over an extended period of time.
The device is expected to remain operational and stable throughout this test and to pass most/all of the legitimate
traffic. If an excessive amount of legitimate traffic is blocked throughout this test, caused by either the volume of
traffic or by the device failing for any reason, this will result in a FAIL.
6.3 Behavior of the State Engine Under Load

This test determines whether the device is capable of preserving state across a large number of open connections
over an extended time period.
At various points throughout the test (including after the maximum has been reached), it is confirmed that the
device is still capable of inspecting and blocking traffic that violates the currently applied security policy while
confirming that legitimate traffic is not blocked (perhaps as a result of exhaustion of the resources allocated to state
tables). The device must be able to apply policy decisions effectively based on inspected traffic at all load levels.
6.3.1 Attack Detection/Blocking – Normal Load
This test determines whether the device can detect and block policy violations as the number of open sessions
reaches 75% of the maximum determined in Test 5.3.1.
25
6.3.2 State Preservation – Normal Load

This test determines if the sensor maintains the state of pre-existing sessions as the number of open sessions
reaches 75% of the maximum determined in Test 5.3.1.
A legitimate HTTP session is opened, and the first packet of a two-packet exploit is transmitted. As the number of
open connections approaches the maximum, the initial HTTP session is completed with the second half of the
exploit, and the session is closed. If the sensor is still maintaining state on the original session, the exploit will be
recorded. If the state tables have been exhausted, the exploit string will be seen as a non-stateful attack and will
thus be ignored. Both halves of the exploit are required to trigger an alert. A product will fail the test if it does not
generate an alert after the second packet is transmitted or if it raises an alert on either half of the exploit on its own.
6.3.3 Pass Legitimate Traffic – Normal Load
This test ensures that the device continues to pass legitimate traffic as the number of open sessions reaches 75% of
the maximum determined in Test 5.3.1.
6.3.4 State Preservation – Maximum Exceeded
This test determines whether the device maintains the state of pre-existing sessions as the number of open sessions
exceeds the maximum determined in Test 5.3.1. The method of execution is identical to Test 6.3.2.
6.3.5 Drop Legitimate Traffic – Maximum Exceeded
This test ensures that the device continues to drop all traffic as the number of open sessions exceeds the maximum
determined in Test 5.3.1.
Note: If a device allows traffic to "leak" due to the way it expires old connections, this will result in an automatic fail
for the entire test.
6.4 Protocol Fuzzing & Mutation

This test stresses the protocol stacks of the firewall by exposing it to traffic from various protocol randomizer and
mutation tools. Several of the tools in this category are based on the ISIC test suite. The device is expected to remain
operational and capable of detecting and blocking exploits throughout the test.
6.5 Power Fail

Power to the device is cut while passing a mixture of legitimate and disallowed traffic. Firewalls should always be
configured to fail closed—no traffic should be passed once power has been cut.
6.6 Backup/Restore
Backing up and restoring a device's configuration is a critical component of deploying any managed device within a
live network. It should be possible to export configurations and store them offline for backup purposes. Additionally,
it should be possible to completely reconfigure the device using the offline configuration file(s). This includes
restoring all policies and interface information in order to deploy a device.
6.7 Persistence of Data

The device should retain all configuration data, policy data, and locally logged data once it has been restored to
operation following a power failure.
26
7 Management & Reporting Capabilities

Some firewalls are managed by a cloud management portal, while other are managed by a central management
server. Either is acceptable for this test. The management system must retain all log and event information about
the malicious activities that enabled systems to achieve conviction events.
The firewall offering should provide the following centralized visibility:
7.1 Authentication
7.1.1 Role-Based Access Control (RBAC)
The system supports RBAC.
7.1.2 Authentication
Third-party authentication systems such as LDAP and Active Directory are supported.
7.2 Policy
7.2.1 Policy Definition
Define and save multiple security policies.
7.2.2 View Policy
When an alert is selected, the firewall provides the ability to access directly (single-click) and view the
policy and rule that triggered the event.
7.2.3 Policy Association
Once policies have been defined, is it possible to apply them to specific users and/or groups.
7.2.4 Policy Inheritance
The firewall allows (by default) the creation of groups and sub-groups such that sub-groups can inherit
certain aspects of configuration and policy definition from parent groups.
7.2.5 Policy Version and Checksums
By default, the firewall records the version and the hash of a policy (ensures policy is not tampered with by
3rd parties).
7.2.6 Bulk Operations
Ability to efficiently search for individual signatures or groups/classes of signatures, and subsequently to
apply one or more operations to an entire group in a single operation (for example, to enable or disable a
group of signatures, or to switch a group from block mode to log mode, etc.).
7.3 Logging
The use of standardized logging and reporting formats, which facilitate the fast and accurate consumption of
presented data, is imperative to enable administrators to assess conviction accuracy. The firewall offering should
allow easy generation and exportation of reports, logs, and/or alerts into one or more of these formats:
o CSV
o XML
o JSON
o Other formats may be acceptable; please coordinate with CyberRatings to ensure compatibility and
adequate capabilities.
27
The following will be evaluated for each firewall offering:
7.3.1 Malicious Traffic
Malicious traffic information from the firewall is logged and displayed centrally.
7.3.2 Administrator Login/Logout
Session login/logout is recorded in the logs.
7.3.3 Successful Authentication
Administrative authentication status is included in the logs.
7.3.4 Unsuccessful Authentication
An administrative authentication attempt is included in the logs.
7.3.5 Policy Change
Policy changes are included in the logs.
7.3.6 Policy Deployed
Policy deployment is included in the logs.
7.3.7 Hardware Failure
Hardware failure detected.
7.3.8 Power Cycle
Something was power cycled.
7.3.9 Log Time Normalization
If multiple time zones exist with the system, logs from multiple sources are correlated using a common
time zone to assist in administrator readability and forensics.
7.3.10 Log File Maintenance
Provides log file maintenance options (automatic rotation of log files, archiving, etc.).
7.3.11 Forensic Analysis
The firewall can capture traffic for later review.
7.4 Alert Handling

7.4.1 Centralized Alerts
All alerts are delivered to, and handled by, a single management console.
7.4.2 Performance Alerting
Alerts can be created to provide visibility into underlying performance and capacity (for example, if disk
quota is close to being exceeded or if utilization is above a specific threshold).
7.4.3 Alert Filtering
The ability to select a particular piece of data from an alert and summarize on that data field (i.e., select a
source IP address and view all alerts for that source IP).
7.4.4 View Alert Detail

The ability to select an individual alert and view the additional information.
7.4.5 View Packet Contents
The ability to select an individual alert and view the contents of the trigger packet or context data for the
exploit.
7.4.6 Alert Suppression
The ability to create exception filters based on alert data to eliminate further alerts that match the
specified criteria (e.g., same alert ID from same source IP). This does not disable detection, logging, or
blocking but merely excludes alerts from the console display.
7.4.7 Incident Workflow
The ability to annotate and track incidents to resolution.
7.4.8 Correlation (automatic)
The means to automatically infer connections between multiple alerts and group them together as
incidents.
7.4.9 Correlation (manual)
The means for the administrator to manually infer connections between multiple alerts and group them
together as incidents.
7.5 Reporting
7.5.1 Custom Reports
The product includes a report generator providing the ability to construct complex data filters in a search
form and summarize alerts on the specified search criteria.
7.5.2 Saved Reports
A custom report filter is available and can be saved for subsequent use.
7.5.3 Report Automation
Support report scheduling and automated delivery mechanisms.
7.5.4 Centralized Reports
Capable of providing summary reporting on all alerts from a single, central management console.
7.5.5 Built-In Reports
Provide built-in reports covering typical requirements such as a list of top attacks, top source/destination IP
addresses, top targets, etc.
7.5.6 Industry Reporting Standards
Support for reporting format standards, for example, Syslog.
7.6 Change Control

It is essential for the system to track, retain, and report changes to policies and rules. User access should also be
monitored and logged, and, if possible, change management controls should be implemented. These items fall
29
under compliance process controls for change management, onboard and off-board, segregation of duties, and
access control. For more information on access controls, see the section on Authentication.
7.6.1 Change Control Logging
Critical during audits of the system to track all details of a change. To score a "yes" in this section, the
firewall must demonstrate that the system contains the user name, date, and time of the changes, as well
as details of the change.
7.6.2 Roll-Back
Roll-back is dependent on maintaining a history of revisions and must allow an administrator to select a
prior security policy and restore it to current operation. To score a "yes" in this section, the firewall must
demonstrate that revisions are automatically saved and that any prior rule/policy can be restored.
7.6.3 Revision History
Revision history is required during any audit, and the data is necessary for the prior two features to work.
To earn a "yes" in this section, systems must perform an automatic backup of the policy/rule set upon a
change and provide the ability to generate an automated differential report between any two revisions.
30
8 Total Cost of Ownership and Value

Implementation of security solutions can be complex, with several factors affecting the overall cost of deployment,
maintenance, and upkeep. All of the following should be considered over the course of the useful life of the
product:
• Product Purchase – The cost of acquisition.
• Product Maintenance – The fees paid to the vendor, including software and hardware support, maintenance,
and other updates.
• Installation – The time required to take the device out of the box, configure it, put it into the network, apply
updates and patches, and set up desired logging and reporting.
• Upkeep – The time required to apply periodic updates and patches from vendors, including hardware, software,
and other updates.
31
Contact Information
CyberRatings.org
2303 Ranch Road 620 South
Suite 160, #501
Austin, TX 78734
info@cyberratings.org
www.cyberratings.org
© 2021 CyberRatings.org. All rights reserved. No part of this publication may be reproduced, copied/scanned, stored on a
retrieval system, emailed or otherwise disseminated or transmitted without the express written consent of CyberRatings.org.
(“us” or “we”).
1. The information in this report is subject to change by us without notice, and we disclaim any obligation to update it.
2. The information in this report is believed by us to be accurate and reliable at the time of publication, but is not guaranteed.
All use of and reliance on this report are at your sole risk. We are not liable or responsible for any damages, losses, or
expenses of any nature whatsoever arising from any error or omission in this report.
3. NO WARRANTIES, EXPRESS OR IMPLIED ARE GIVEN BY US. ALL IMPLIED WARRANTIES, INCLUDING IMPLIED WARRANTIES
OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE, AND NON-INFRINGEMENT, ARE HEREBY DISCLAIMED AND
EXCLUDED BY US. IN NO EVENT SHALL WE BE LIABLE FOR ANY DIRECT, CONSEQUENTIAL, INCIDENTAL, PUNITIVE, EXEMPLARY,
OR INDIRECT DAMAGES, OR FOR ANY LOSS OF PROFIT, REVENUE, DATA, COMPUTER PROGRAMS, OR OTHER ASSETS, EVEN IF
ADVISED OF THE POSSIBILITY THEREOF.
4. This report does not constitute an endorsement, recommendation, or guarantee of any of the products (hardware or
software) tested or the hardware and/or software used in testing the products. The testing does not guarantee that there are
no errors or defects in the products or that the products will meet your expectations, requirements, needs, or specifications,
or that they will operate without interruption.
5. This report does not imply any endorsement, sponsorship, affiliation, or verification by or with any organizations
mentioned in this report.
6. All trademarks, service marks, and trade names used in this report are the trademarks, service marks, and trade names of
their respective owners.
32

Test Methodology: Enterprise Firewall

Uploaded by

Document Information

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Test Methodology: Enterprise Firewall

Uploaded by

Copyright:

Available Formats

TEST METHODOLOGY

2 Access Control ............................................................................................................................... 7

3 SSL/TLS Functionality .................................................................................................................... 8

3.2 Cipher Selection .............................................................................................................................8

3.4 Decryption Bypass Exceptions ........................................................................................................9

4 Threat Prevention ....................................................................................................................... 11

4.2 Exploits ........................................................................................................................................ 11

Enterprise Firewall v1.0

5.1.3 256 Byte Packets ................................................................................................................................................... 19

5.2 Latency ........................................................................................................................................ 20

5.3 Maximum Capacity ...................................................................................................................... 20

5.4 HTTP Capacity .............................................................................................................................. 21

5.5 Application Average Response Time: HTTP .................................................................................. 22

5.7 Application Average Response Time: HTTPS................................................................................. 23

6 Stability and Reliability ............................................................................................................... 25

6.4 Protocol Fuzzing & Mutation........................................................................................................ 26

Enterprise Firewall v1.0

7 Management & Reporting Capabilities ....................................................................................... 27

7.2 Policy ........................................................................................................................................... 27

7.3 Logging ........................................................................................................................................ 27

7.4 Alert Handling.............................................................................................................................. 28

7.5 Reporting ..................................................................................................................................... 29

7.6 Change Control ............................................................................................................................ 29

8 Total Cost of Ownership and Value.............................................................................................. 31

Enterprise Firewall v1.0

1.2 About This Test Methodology

• Security effectiveness • Stability and reliability

• Packet filtering • Integrated intrusion prevention system (IPS)

1.3 Inclusion Criteria

Enterprise Firewall v1.0

1.4 Product Guidance

Enterprise Firewall v1.0

2.1 Baseline Policy

2.2 Simple Policy

2.3 Complex Policies with Application Control

• Popular social networking Web sites (Web applications)

Enterprise Firewall v1.0

3.1 Decryption Validation

3.2 Cipher Selection

3.3 Cipher Support

Enterprise Firewall v1.0

3.3.2 Prevention of Weak Ciphers

• Null ciphers (no encryption of data provided)

3.4 Decryption Bypass Exceptions

• Layer 3 information (i.e., bypass based on source or destination IP address)

Enterprise Firewall v1.0

3.5 Certificate Validation

3.6 TLS Session Re-use

Enterprise Firewall v1.0

4.1 False Positives

Enterprise Firewall v1.0

• A reverse shell is returned

4.3 Protection Resiliency

Enterprise Firewall v1.0

Enterprise Firewall v1.0

Enterprise Firewall v1.0

• Fragments from 8 to 32 bytes in size

• Segments from 1 - 2048 bytes in size

Enterprise Firewall v1.0

Enterprise Firewall v1.0

4.4.4 HTTP Compression

Enterprise Firewall v1.0

4.4.5 HTML Obfuscation