You are on page 1of 22

Larry Suto – November 2011

Analyzing The Effectiveness of Web Application Firewalls

Analyzing the Effectiveness of Web Application Firewalls
A benchmark study of eight web application firewalls and intrusion prevention systems

by Larry Suto Application Security Consultant San Francisco November 2011

1

Larry Suto – November 2011

Analyzing The Effectiveness of Web Application Firewalls

Table Of Contents
Analyzing the Effectiveness of Web Application Firewalls Table Of Contents 1. Executive Summary 2. Key Findings Overall Findings Findings: Baseline Tuned Findings: DAST/Scanner Tuned 3. Evaluation Criteria & Results Evaluation Notes Barracuda Networks: Barracuda 660 Citrix: NetScaler NS9.3 Build 48.6.nc DenyAll: rWeb 4.0 - Core manager v0.3.0.220 F5: ASM 3900 Imperva: SecureSphere 8.0 Trustwave: ModSecurity 2.6.2 SourceFire: SourceFire 3D Sensor 2100 - 4.9.1.6 build 150 Unnamed IPS 4. Conclusions 5. Challenges False Positive Risk WAF Evasion 6. Background 7. Methodology Overview Constraints and Assumptions Targets of Attack (TOA) Testing Tools Experimental Tests Setup TEST BL — Baseline Test Vulnerabilities Included Configuration Notes 8. Future Directions 9. References 10. Bibliography 11. Acknowledgements Appendix A: Definition of Terms and Acronyms Appendix B: Test sites and vulnerability data

2

The other three platforms are not yet supported the DAST tool used in this study. The tests were as follows: 1.Larry Suto – November 2011 Analyzing The Effectiveness of Web Application Firewalls 1. Executive Summary This study evaluated the effectiveness of Web Application Firewalls (WAFs) and Intrusion Prevention System (IPS) devices at protecting web applications against external attacks in comparison to each other. In addition. (in alphabetical order): ● WAF’s: Barracuda 660. Each of the tests leveraged the devices in different configurations. There are other DAST vendors that have WAF integration such as Cenzic and Whitehat Security but they were not used due to time and logistical considerations.the % vulnerabilities found with baseline tuning versus the % vulnerabilities found when tuned with DAST generated filters.The NTODefend DAST tuning solution was used in this study to generate filters for five of the eight tested platforms. F5 ASM. ModSecurity ● IPS’s: Sourcefire 3D. The study examined the following cross-section of modern WAFs and IPSs both proprietary and open source. The following graph (Figure 1) shows a high-level summary of the test results . DAST tools perform automated web application vulnerability scans. Figure 1 3 . Imperva SecureSphere. and factors affecting the effectiveness of web application protection solutions. Citrix NetScaler. Future testing should compare these solutions as well. Some of the DAST solutions now generate filters which can be imported into WAF and IPS protection policies. this study provides general guidelines about the ease of use. Baseline Tuned: Tuned configuration requiring 1 day or less of an experienced expert’s time 2. Unnamed IPS Each of the eight systems was evaluated in two separate tests. DAST : Trained solutions by application generated filters from Dynamic Application Security Testing (DAST) software products. DenyAll rWeb.

or F5 Figure 3 Additional general conclusions from this study are as follows: ● ● A web application firewall is a recommended element of the security defense strategy. there was an average improvement of 36% more vulnerabilities blocked when the DAST generated filters were applied.5 hours by a informed user or expert. (see Figure 3 below) IPS solutions are not designed for and are not very effective at blocking web application vulnerabilities with their default settings. The study found that basic tuning takes an average 3. Care should be taken that the tool has a quality ruleset and that false positives have been remediated. Key Findings Overall Findings The most significant conclusions of the study are as follows: ● To maximize effectiveness. WAF solutions must be tuned by a trained professional. However. ● ● ● 4 . Rule creation requires detailed knowledge of both the application and the WAF or IPS solution. when tuned by DAST tool generated filters. and a deep consistency across all rules to ensure that they do not violate each other. They need to be accompanied by solutions & processes such as DAST and Static Analysis Security Testing (SAST) tools. Figure 2 2. (see Figure 6) For the solutions supported. (see Figure 3 below) ● ● NTODefend does not yet support for Barracuda. WAF and IPS solutions are both capable of providing effective protection. risk management etc. but neither solution is a sufficient application defense strategy in isolation.Larry Suto – November 2011 Analyzing The Effectiveness of Web Application Firewalls The following table shows the same results of both tests in a table format and includes both the actual numeric results as well as the percentages. several target systems such as SQL databases. It is not practical for a general user to produce effective filters in a reasonable amount of time. Using an automated solution made the rules generation part of the experiment manageable. IPS solutions can be as or more effective than several of the WAF’s. etc. Organizations should understand and test the differences of their solution’s detection and blocking effectiveness before and after training with DAST generated filters. Training with a DAST tool as only as effective as the tool’s output. otherwise any manual attempt at generation of such rules would require a deep understanding of regular expressions. Citrix.

however. ● 3.g. dependent on the quality of the DAST tuning system. ● Findings: DAST/Scanner Tuned Test two yielded the following results when WAF and IPS solutions were trained by DAST generated filters. The key findings from the first test where the solutions were tuned by an experienced expert were as follows: ● The WAFs are fairly effective at detecting and defending web attacks on their own. Three criteria were defined by which to score the solutions. (See Figure 3 below) ● The IPS solutions improved by an average of 60% bringing up their performance at-par or better than the trained/configured WAFs. The improvement of the WAFs and IPS solutions is. Solutions that only allow DAST solutions to turn on existing proprietary WAF rules (e. In this study.g. then the weighting and specific scoring guidelines were defined for each criteria. with their overall blocking effectiveness averaging 82%. The following table details the evaluation criteria and their weightings: Figure 4 5 . the supported WAF’s that were tuned with DAST solution improved an average of 19% from their baseline tuned state. it became clear that considerable vendor-specific knowledge and understanding is required to properly configure and tune the WAF and IPS solutions.Larry Suto – November 2011 Analyzing The Effectiveness of Web Application Firewalls ● DAST tuning solutions can significantly improve the effectiveness of WAFs if the WAFs allow the DAST to create new customer rules (e. Sourcefire and ModSecurity). the most effective solution found 88% of the vulnerabilities known in the test applications while the average effectiveness across all WAF solutions was 62%. results and information were captured according to the study’s methodology. Evaluation Criteria & Results Throughout the study. The IPS solutions are not designed to protect web applications and were not very effective at defending them during this test. Findings: Baseline Tuned From the first test. Imperva) do not benefit from such improvement.

With some work. The signature strategy seems to be focused mostly on regular expressions which sometimes are difficult to get right.Larry Suto – November 2011 Analyzing The Effectiveness of Web Application Firewalls The results closely matched the results from the percentage blocked scores Figure 5 The following table details the summary of the results from both tests: Figure 6 Evaluation Notes List of the defensive products used and experiences using them for the testing. The default signatures/basic tuning processes were fairly effective but the strict mode did not seem to improve blocking accuracy. Cons: Advanced tuning and signature enablement is a bit counterintuitive and not easy to enable or execute. Basic signatures are easy to enable. advanced signatures can be added. Learning mode is not recommended out of the box because of performance issues. Support: Barracuda’s support is very responsive and adequately addressed all operational issues. ● ● 6 . Barracuda Networks: Barracuda 660 ● Pros: The WAF is fairly full featured and has a responsive and well laid out web GUI. It offers above average protection against attacks. They assisted in turning on all the default necessary signatures and demonstrated how to enable and apply the stricter signatures as well.

Accuracy of blocking was decent in the tests. Nevertheless. Policy configuration and management could be better.The recommended implementation is to allow the appliance to collect traffic to develop and application profile before stricter rules are applied. after turning on a reasonable amount of the policies. Support: Did not use anything other than the projects forums and mailing lists. Cons: The Imperva UI while providing a lot of information is a bit difficult to navigate. the setup can be quite challenging. there are a lot of options for tuning the protection policies. Support: DenyAll support was very helpful in providing instructions to install a custom blacklist using the command line interface. Cons: Management interface could be a little more intuitive. An expert user can modify filters to accomplish nearly any task Cons: Limited only to Apache users. It is also one of the easier WAFs to configure and provides the most value straight out of the box. ● 7 . Live support is purchased separately from 3rd party consulting companies. ● ● Imperva: SecureSphere 8.6. and the default blacklist is easy to configure. Modifying existing policies to handle exceptions can be unwieldy.3. It is supposed to do well against HTTP resource exhaustion attacks. and is installed on the application server.3 Build 48.0 . Support: Relatively little support was required to perform the setup.Larry Suto – November 2011 Analyzing The Effectiveness of Web Application Firewalls Citrix: NetScaler NS9. There is no user interface.Core manager v0. Great intuitive and snappy web based interface.2 ● ● Pros: Free open sourced software makes it lowest acquisition cost.nc ● Pros: The evaluation version of the Netscaler platinum is full featured and includes the web application firewall feature. With basic tuning F5 performed very well in the tests.220 ● ● ● Pros: The web interface is easy to use. Easy to integrate with Apache.0 ● ● Pros: Imperva has a high quality protection engine with a very robust set of basic policies. it became clear that it was a very effective solution. Thus there is a bit of a grey area between what a default policy and a stricter applied policy might be. it does not have a method to optimally import a custom blacklist such as from NTODefend. Once basic configuration is accomplished. The setup was straightforward. Cons: While the interface does allow for custom rules. System can be be chatty with false positives during learning. Requires command-line activity and numerous config files. Cons: For users not familiar with the F5 product line. Support: Imperva was very helpful in assisting with setting up the appliance and ensuring that everything was properly configured.6. Support: F5 was very helpful and gave a comprehensive walk-through of the tool including the basic setup and policy configuration.0. ● Trustwave: ModSecurity 2. The F5 sits on a very advance load balancing platform and provides other advanced security features such as DDOS protection. thereby limiting scalability options. Creating custom policies can be difficult. F5: ASM 3900 ● Pros: Excellent appliance quality. ● ● DenyAll: rWeb 4.

is addressed by "well-trained" defensive tools like WAFs and IPS’s where a vulnerability can be effectively patched while the code is being repaired. The IPS solution supports Snort 2. In particular. ● Support: Their support team was very helpful in overcoming the few configuration difficulties encountered. However. This problem. however they take considerable amount of time: days.6 compliance. several conclusions can be made without doubt. ActiveX and Flash plugins. This study clearly demonstrated that using WAFs/IPS devices in default or minimal mode to protect web applications are often not useful. For instance. This has been well documented (WAF Evasion section in this document). but not fixed at the source. appealing. external service etc. and visually oriented.9. For some organizations. Manual code reviews and code fixes are essential. it is almost certain that even with a reasonable amount of learning and tuning most WAFs will miss well-know and well understood vulnerabilities. SourceFire supports the SourceFire/NTODefend solution as a WAF for PCI section 6. which was used for this testing. and this data may not speak to all of the various claims and may not be fair to some of the individual strengths of each technology. and importing/enabling custom rules is easy to accomplish. ● ● Unnamed IPS This IPS vendor did not give authorization to be named in this study. It is also important to note that while the tests are complete as far as they go. Additionally.4. What is interesting is that training these tools with a DAST filter generator can dramatically increase their effectiveness and make them a far more useful part of an enterprise's application security strategy. weeks or even months once a vulnerability becomes known to the developers. it is the fairest and most impartial model that the study could come up with to test these technologies in a vendor-neutral way. Challenges The WAF vendors have spent a significant amount of time making claims of one sort or another as to why their technology is better. legacy software. trained IPS devices can be effective at protecting Web Applications. where for a period of time a vulnerability is known. contractual issues and elaborate change control processes means that known vulnerable applications can and do long periods without being remediated. 4. 5. ● Pros: The UI was very compact. ● Cons: Not useful on its own to function as a WAF. In case where users do not have access to source code for any number of reasons. and WAFs configured with the recommended policies and tuned can often be bypassed by a moderately skilled hacker or an automated hacking tool.6 build 150 ● Pros: The web interface is very full featured. but their support team was able to clear up the problem quickly.1.Larry Suto – November 2011 Analyzing The Effectiveness of Web Application Firewalls SourceFire: SourceFire 3D Sensor 2100 . due to time constraints and to keep the scope of the study reasonable. (3rd party software. Even with this limited level of testing. Cons: Not useful on its own to function as a WAF. 8 . Support: Sourcefire was very helpful when needed. Suffice it to say that the problem escalates polynomially once we start throwing in complications due to AJAX/JSON type sites or various Java. Conclusions WAFs are a controversial technology. others argue the opposite [6]. WAF evasion research was completely left out. And finally are we staring at the abyss of a flawed model at detecting attacks? If WAFs could be made more intelligent somehow the future is bright for this technology. that this test is not an exhaustive study and only begins to scratch the surface of the problem. Had some troubles with clearing rules and reloading them.) there is no manual recourse for fixing bugs.8 rules. While some security professionals argue that they are easily avoided and the only remediation to web application vulnerabilities is to fix the code [7]. There is also a very active community of support around the snort project which includes volunteers as well as Sourcefire paid staff.

I think that a defense 9 . The details are here "http://code. In this case -treplaceComments altered the input (by ignoring standalone termination */ and simultaneously replacing unterminated /* comments with a space( 0x20)) so that the filter could not recognize it as an attack. where the upper bound of successful detection seems unable to cross the 60% mark. but in the transformation function -t:replaceComments (Modsecurity transformation functions are a pipeline of normalization functions that modify the input so that the rule can detect the attack)." One of the attacks presented bypassed "ALL" the PHPIDS rules as of 0. an individual by the name of Johannes Dahse used a combination of MySQL comments and new line characters to evade the CRS SQL Injection filters. In the end it was not possible to create a rule to remove SQL comments reliably so another rule with a complex regular expression was created instead to detect the presence of SQL comments themselves. Example 3: JSReg issues has had issues as well. the following examples were retrieved from the Web: ● Example 1: ModSecurity SQL Injection challenge held by Trustwave's SpiderLabs.google. this was a subset of the flawed websites used in the current study. Not notice that one of them is a regular expression rewrite error: "the failure to strip the single line comment which then fools the regex rule into thinking that the code is a regex object and not function calls".Larry Suto – November 2011 Analyzing The Effectiveness of Web Application Firewalls False Positive Risk Although there was not time to complete a comprehensive study of false positives (the blocking of good traffic) A limited amount of effort was used to try an eliminated false positives. but in future tests this area should have more attention. ● Example 2: The second example comes of failed context-free matching techniques comes to us from a paper from the recent Defcon 19. The primary focus in this study was blocking of known vulnerabilities.5. We have already seen that pattern matching does not yield optimum results elsewhere in the industry.{2. At the time of the writing of this paper. In this challenge the ModSecurity WAF was tuned and set up to protect a set of four of the scanner vendor vulnerable websites.php (?:(. The problem was not in the actual filter itself./lib/IDS/Converter. How could of this happened you ask? The flaw was the incorrect assumption that "attacks can never be repetitive". So lets take a look at what happened. if it is repeated 33 times it will bypass PHPIDS entirely.6. Coincidentally. WAF Evasion After a cursory review it can be discovered that there are numerous examples in the field where it becomes obvious that the status-quo solutions to run-time attack detection and containment technology are routinely by-passed. In one of the successful bypasses of the ModSecurity inbound CRS SQL Injection rules. I refer to Anti-virus applications of signature detection. Even though the attack string “<script>alert(1)</script> is matched by 4 filter sets.6.The first match is a back reference created with the \1 and then 32 additional matches are required to trigger the regex. }) \1{32. which used regular expressions by the way. ● The commonality between these examples is that all them are using pattern matching—in particular regular expression type of context free matching—with a little bit of AI and boundary restriction of parameter values thrown in to detect attacks. The talk discussed the paper titled "Bypassing PHPIDS 0.}) The regular expression matches any two characters which are repeated 33 or more times.com/ p/jsreg/wiki/Exploits".5. The heart of the problem was the following bad expression: Line 90 in .

There is a list of do’s and don’ts. and their core approach of staying that way is to keep out of the comparative-studies field all together. “DON’T think it’s set and forget” [1]. It remains difficult to find standards that provide direct comparative quantitative measures. These difficulties arise from the inability to compare apples to apples when it comes to comparing Web Application Firewalls (WAFs). The most promising guide for end-users seems to conduct such a study seems to be The Web Application Firewall Evaluation Criteria (WAFEC) and its corresponding WAFEC Evaluation Response Matrix. and presents varying philosophies of how Web attacks should be defended against [2]. 10 . it falls on to the consumer to fund an internal study on some standards basis. There have been some publications discussing the functionality of WAFs but most have been limited to discussions on defining WAF functionality and providing advice on what keep in mind when conducting private evaluations and deployments. with advice like “DO consider WAF for performance monitoring”. They still require extensive knowledge of each aspect of web technology and this author found that Evaluation Response Matrix has no mention of relative importance of each criterion. 3]. Security pundits are at odds with each other on whether the problem can be solved using fully automatic techniques such as WAFs and IPSs or is it only solvable using partial automated tools such as staticanalysis tools in conjunction with manual code review and remediation strategies. So. and especially difficult to find ones that can explain it to average users of this technology. and similar defense technologies that purport to safeguard against Web attacks [1]. however they are very sensitive to vendor-neutrality.6. The consensus is that a WAF is an HTTP focused implementation of general purpose IPS technology. network Intrusion Prevention Systems (IPSs). I think that next section discusses some future directions that have promise. This study is also one of the first to provide clear empirical evidence that without specialized configuration and tuning. However. PCI DSS 6. 6. WAFEC only provides only qualitative comparative analysis between potential candidate solutions.Larry Suto – November 2011 Analyzing The Effectiveness of Web Application Firewalls mechanism that can only protect against 70-80% of the attacks is like a house that’s missing a wall. Each Web application firewall vendor delivers different strengths and weaknesses. 2. Background When comparing web application security solutions amongst each other for the purpose of assessing which one to choose and why consumers are faced with several difficulties.6]. as things stand. Security certification standards such as Common Criteria. To the best of the author’s knowledge this is the first study to conduct a general purpose evaluation of WAF and IPS systems’ ability to protect web applications using a uniform testbed with a simple and easily reproducible methodology. DIACAP etc. We have a long way to go. Non-profit consortia provide an invaluable resource to the consumer. Previous research on WAFs have been confined to vendor organizations and customer evaluation engagements. and while the state-of-art against attackers is not where we need it to be. a general purpose IPS provides limited protection against web application attacks. focus on the theoretical for the most part. An interesting thing to keep in mind is that in the literature a WAF is referred to as security mechanism with an intimate understanding of the nature of HTTP traffic and web application vulnerabilities. nor any method of how to measure evaluate over-all effectiveness of the targets of evaluation [5. See the appendix for some references. and do not provide a reasonable guideline for comparison. despite their scorecard nature [1.

Larry Suto – November 2011 Analyzing The Effectiveness of Web Application Firewalls 7.http://testphp.com Acunetix TestPHP .vulnweb.vulnweb. Methodology Overview The study tested each solution against the same set of websites and web application prototypes to ensure that the experiments were instantiated against well-known and well understood vulnerabilities. Targets of Attack (TOA) The study is a comparative analysis of primary results from a set of controlled tests conducted on a collection of vulnerable site.http://www. This study is not an exhaustive dissertation of various advanced capabilities of the evaluated web application firewalls and intrusion prevention systems. while knowledgeable about the Web application security principles. The primary DAST products (vulnerability scanners) used were NT OBJECTive’s NTOSpider and Mavituna’s Netsparker. 11 .http://testaspnet.http://demo. The user can detect vulnerabilities within the security target. The user does not fully understand the underlying technological details such-as deep understanding of HTTP protocol negotiations.http://zero. and stand alone applications such as ThreadFix (previously called Vulnerability Manager) from Denim Group. The user only has limited time and budget to perform these evaluations.webscantest. The vulnerable site used is called Target of Attack (TOA). reporting and thwarting web attacks. The following TOAs were used in these experiments: ● ● ● ● ● ● Acunetix AspNet .http://crackme. This provided a uniform baseline for comparison. There are other DAST vendors that have WAF integration such as Cenzic and Whitehat Security.com Cenzic Crackme . The user.com IBM Testfire . Constraints and Assumptions The experiments in the study were designed with following constraints and assumptions: ● ● ● ● ● ● A typical end-user (hereinafter user) wants to protect a vulnerable web application or website.com These sites demonstrate manifestations of well known and well understood vulnerabilities. using an automated tool. NT OBJECTive’s NTODefend product was used for filter generation. is not familiar with any vendor specific details for the products being evaluated. The purpose of this study is to create a foundation for such comparative studies.net NTO Web scan test .webappsecurity. hereinafter called the security target.cenzic. This experimental study provides a quantitative comparison between various web application security solutions that include both Web Application Firewalls and Intrusion Prevention Systems and evaluates their relative effectiveness in detecting.testfire.com HP zerowebappsecurity . Testing Tools The study required use of at least one DAST product capable of creating filters for as many as possible of the above listed defensive products.

In this test the WAF/IPS is configured to protect the application with approximately of one half day of effort. TEST I Scanning the security targets with a WAF/IPS in front of the vulnerable applications. Vendor requested assistance is provided as necessary to ensure that the recommended protection policies are in place. A third-party rule generator is then used to import policies in order to improve the blocking capability of the system. This is the Tuned/Trained mode.Larry Suto – November 2011 Analyzing The Effectiveness of Web Application Firewalls NT OBJECTive’s NTODefend product includes a QuickScan feature which tests for false positives (the blocking of good traffic) by sending both legitimate and attack payloads to confirm that only attack traffic is being blocked this was only used on two sites as a sanity check but expanding false positive would be important in future tests. The other tools and features were not used due to time and logistical considerations. In this test the WAF is configured to protect the application with a minimum of one half day effort. as shown in Figure 1 to detect exploitable vulnerabilities in the TOAs. Figure 7 Figure 1 (above) depicts the WAF setup as it relates to the scanner and protected application location for Tests I and II. The study is comprised of the following tests: TEST BL — Baseline Test A security scan the security targets of vulnerable systems for bugs using two These Scanners are the used for subsequent tests for each of the Evaluation Target WAF or IPS. For the purposes of this report. and which most users care about. Vulnerabilities Included Each WAF/IPS is different in its specific methodology for finding and preventing attacks. These are: ● ● ● ● ● ● SQL Injection / Blind SQL Injection Cross Site Scripting / Persistent Cross Site Scripting Command Injection CSRF / HTTP Response Splitting Arbitrary File Upload attacks Remote File Include (PHP Code Injection) 12 . Future testing should compare these solutions as well. TEST II Scanning the security targets with a WAF in front of the vulnerable applications. Experimental Tests Setup The evaluation tests used the DAST scanners independently. Vendor requested assistance is provided as necessary to ensure that the recommended protection policies are in place. the types of vulnerabilities that were counted were those that are useful against custom applications. This is the DAST or Scanner Trained mode.

So what does it all mean? Is run-time automated defense doomed to failure? I don’t think so. and will probably reach an upper bound that is well below the required threshold of risk mitigation. To then see how well customized training from an outside source would improve the protection. there is a strong debate on whether or not programmers can be taught to create flawlessly secure programs [6]. artificial intelligence and other related areas that may prove to be much more powerful weapons in fight against attacks. very important issue of protecting Web Application Assets.Larry Suto – November 2011 Analyzing The Effectiveness of Web Application Firewalls The vulnerabilities were recorded to cross reference which WAF's blocked which vulnerabilities in a simple format to track and compare the results. mathematics. For many of the systems. 8. A complete listing of vulnerabilities blocked by each WAF was recorded to track the effectiveness of each technology. The collected data is being made freely available for anyone to review and re-create. The assumption was that protection would improve with training. Again.matching systems. and this was clearly proven during testing. This resulted in a fairly high degree of confidence that the extent of Blocked and Missed numbers are complete. 13 . I received direct assistance from the vendor in order to accomplish this. Even today. it should be understood that more manual configuration effort could likely be used to improve the results of the WAF's further. As we have seen. but in approaches that fundamentally redefine the problem and by doing so discover innovative approaches to solving this very real. There are entities and companies doing research in interesting directions in computersciences. this was done mostly when it was obvious that the basic tuning process of the WAF was going to involve a significant effort. I think that the true solutions lies not just in ever intractable construction of more and more complex pattern. Future Directions So what are the future directions? Surely we cannot do manual defense. automated defense systems that perform strictly signature or pattern based detection only achieve partial success at best. as well as when compiling a study like this. Configuration Notes For tuning and training the basic policy was enabled plus any settings that were recommended by the vendor in order to optimally configure the system in a reasonably effective protection mode. Additionally. custom filters/rules were generated using the DAST tuning solution for each supported WAF and loaded them in and repeated the test. but time constraints are always going to be an issue in real world use.

org/f/ WAFEC_Evaluation_Response_Matrix_1. Special thanks to: Kasey Cross at Imperva Ido Berger and Tom Spector at F5 Networks Grant Murphy at Barracuda Networks Renaud Bidou at DenyAll Also I would like to thank NT OBJECTives and Mavituna Security for DAST scanner and filter support.org/f/wasc-wafec-v1. http://www.0. 2006. 2011-10-25. Implement. (2010) Common Criteria – a good concept in transformation http://community. In addition I would like to thank Amir Khan for helping host the systems and Ahmed Masud for valuable editorial commentary. 2011-07-25. Web Application Firewall Evaluation Criteria (WAFEC) 1. Bibliography Brian Soliman Blog. Technical Articles.webappsec.aspx F5 Networks and WhiteHat Security Solutions (2008). Mary Brandel. Robert Lemos. Vulnerability Assessment Plus Web Application Firewall (VA+WAF) 11.com/2011/07/25/common-criteria-cc-for-informationtechnology-security-evaluation/ [4] WebAppSec. The more scrutiny we place our solutions under.wordpress. 2006. J. Things always seem to take longer than you expect. Copyright 2009 Burton Group. http://projects. 2009-06-11. http://projects.org. This is a nascent space with a lot of potential.au/article/ 307044/web_app_firewalls_how_evaluate_buy_implement [3] Brian Soliman Blog. 14 . WAFEC Response Matrix. Acknowledgements Thanks to all the WAF and IPS vendors for all their graciousness and patience. Ramon Krikken.com. http://www.Larry Suto – November 2011 Analyzing The Effectiveness of Web Application Firewalls 9.com/vulnerability-management/167901026/security/attacks-breaches/231901651/time-toautomate-web-defenses.xls [6] OWASP AppSec USA 2010 Keynote. Presentation at InfoSec 2009 [2] Web App Firewalls: How to Evaluate.ca.html 10. References [1] Web Application Firewalls. What’s in a name?. Application Security — It’s a jungle out there [7] Time to Automate Web Defenses?.org. 2011-07-25.webappsec. the better for everyone. Buy.pdf [5] WebAppSec. Project Coordinator: Ofre Shezaf.darkreading.wordpress. Technical Articles.0.com/blogs/iam/ archive/2010/03/25/common-criteria-a-good-concept-in-transformation. http://bryansoliman.0.com/2011/07/25/common-criteria-cc-for-information-technology-securityevaluation/ Brickman. Common Criteria (CC) for Information Technology Security Evaluation http://bryansoliman.cio. Common Criteria (CC) for Information Technology Security Evaluation.

The program is then monitored for unanticipated behavior. OS Command Injection .Fuzz testing or fuzzing is a software testing technique that applies random unexpected data to inputs of a computer program. which is then interpreted by the target as two HTTP responses instead of one response.Distributed Denial of Service. port number) for purpose of resource access control. This variation uses multiple clients to perform attacks: See also Denial of Service Attack Fuzzing . Origin .An attack that simply tries all possible permutations of a given scenario to discover and exploit any possible vulnerabilities. HTTP Response Splitting . See Same-origin context Same-origin context .SQL injection attack utilizes the lack of input validation vulnerability of web data that an application uses to build a SQL query string. input.An attacker's ability to send a single HTTP request that forces the web server to form an output stream. An attacker can use such a vulnerability construct an input in such a way as to execute arbitrary code on the back-end database. Cross-site scripting .See Deny-by-default XSS .Denial of Service: See also Denial of Service Attack DDoS . application layer protocol. will result in rejection of such an authorization.The ability to include executable code (typically Javascript code) in non Sameorigin contexts. DoS .See Cross-site Scripting 15 .An authorization paradigm where conditions. An attacker can use such a vulnerability to execute arbitrary programs on the host operating system.Larry Suto – November 2011 Analyzing The Effectiveness of Web Application Firewalls Appendix A: Definition of Terms and Acronyms Blacklist .See fuzz testing Fuzz Testing .An injection attack that utilizes the lack of input validation on web applications that execute out to command line applications. properties and objects across different domains (origins). output and processing actions must be explicitly allowed (using a Whitelist). that when matched by a pattern matching system that decides to authorize an operation. methods. SQL injection attack .Any attack that affects the availability and normal usage of a service. This type of vulnerability can be exploited to perform several web application based attacks. Resources (objects.The list of items. Brute-force attack . See also: Origin Deny-by-default . The rationale behind deny-by-default paradigm is to significantly reduce the possibilities of unexpected or malicious behavior. Denial of Service attack .A programmatic boundary in browser-side programming languages that restricts access to methods. instances) can access other resources created from the same origin.Defined as the tuple (domain name. Whitelist .

Larry Suto – November 2011 Analyzing The Effectiveness of Web Application Firewalls Appendix B: Test sites and vulnerability data Summary of raw results 16 .

vulnweb.Larry Suto – November 2011 Analyzing The Effectiveness of Web Application Firewalls Acunetix AspNet .com 17 .http://testaspnet.

http://testphp.vulnweb.com 18 .Larry Suto – November 2011 Analyzing The Effectiveness of Web Application Firewalls Acunetix TestPHP .

Larry Suto – November 2011 Analyzing The Effectiveness of Web Application Firewalls Cenzic Crackme .cenzic.com 19 .http://crackme.

Larry Suto – November 2011 Analyzing The Effectiveness of Web Application Firewalls HP zerowebappsecurity .com 20 .http://zero.webappsecurity.

http://demo.Larry Suto – November 2011 Analyzing The Effectiveness of Web Application Firewalls IBM Testfire .testfire.net 21 .

Larry Suto – November 2011 Analyzing The Effectiveness of Web Application Firewalls NTO Web Scan Test .com 22 .http://www.webscantest.