You won’t believe that blind can see: Benchmarking SQL injection scanners

Andrew Petukhov, Karim Valiev Moscow State University

Thursday, May 24,

Results
• Our approach showed that there’s no such thing as ‘the best scanner’ • Questions? • See you!!!

Thursday, May 24,

Motivation
• Pentesting and auditing
p0wn vs fight for completeness good idea to collect the low hanging fruits first but what if a scanner finds nothing? are there really no vulns? this is a point when you want to know about limitations of your tools and what about of a superposition of tools? I.e. skipfish then w3af then sqlmap

• Look if we could do better than Larry Suto • Look if we could produce a sound testing

Thursday, May 24,

SQLi detection methods
• Error-based
a baseline method, should be good in every scanner should be no problems with implementation

• Blind time-based
main problem: submit a payload which leaves a query syntactically correct SELECT id, date, text FROM news LIMIT ?, 1

• Blind content-based
main problem: compare “true” and “false” responses http://vulnapp/item?id=13 should be the same as http://vulnapp/item?id=13 and 1=1 and differ from http://vulnapp/item?id=13 and 1=0 what is “the same” for an automated tool?

Thursday, May 24,

What our research IS NOT
• It is not sponsored => Acunetix does not win :(((( • It does not give an answer on stupid questions like “what is the best scanner?”
suppose you have two equal scanners implementing time-based technique scanner A: if(substr(field,i,1) = ‘a’, sleep(5), 0) for all characters scanner B: if(substr(field,i,1) < ‘n’, sleep(5), 0) - dichotomy one minimizes scan time and the other - number of requests - which one is better?

• It does not measure the “crawling process”
yes, we know that in point-and-shoot scenario it is crucial

• Quantity metrics is not a final result • We do not expect tools to test for second order SQLi

Thursday, May 24,

Wait! Tell me more about metrics!
• One could expect to measure scanners using FP/FN rates • Imagine: scanner A detects 80 vulns out of 100 and scanner B detects 20 out of 100
Scanner A is better? You cannot tell! 80 test out of 100 were error-based and 20 out of 100 were blind If we generated 100 000 blind tests and 1000 error-based, FP/FN rates would change!

Thursday, May 24,

Wait! Tell me more about metrics!
• Now one could expect to measure scanners using FP/FN rates on test case classes
Like (80%, 20%, 30%) for coverage and (0.1%, 0.9%, 0.2%) for FP

• Scanner metrics become incomparable
consider (80%, 20%, 30%) vs (55%, 85%, 0%) you can hope that one result would inset the other {(80%, 20%, 30%) could include (60%, 10%, 25%)}, but this does not happen

• After all, who would establish these classes?
one needs good performance in injection after the LIMIT statement and the other one - good performance in injection into DML queries

Thursday, May 24,

Ideas behind SQLi bench
• We wanted our test cases to be representative • We wanted our test cases to be as complete as possible • How would we do that?

Thursday, May 24,

Our approach

Thursday, May 24,

Classification!
• A model of a general workflow with DBMS interaction
Get user input Validate user input Construct a query Perform a query and handle the result Construct and issue an HTTP response

• Classification of each step • Test set is a permutation of all classes with each other (44 536 test cases)

Thursday, May 24,

Numbers
Vulnerable Vuln. with False positive error output No err. output and no sleep()

Total arachni-0.3 sqlmap-0.9 sqlmap-r5059 wapiti-2.2.1 skipfish-2.03b skipfish-2.06b w3af-1.1 Burp Suite Pro 1.4.07

28848 15088 12246 10939 11280 18110 22142 21972 13459

15688 0 1208 83 0 1680 3360 450 294

23544 13876 10771 10152 10068 16038 19062 21576 13099

2652 0 447 102 0 1036 1540 193 180

Thursday, May 24,

Some findings
• Error-based method - approximately equal
results

• ‘Blind’ method - all scanners perform bad • Bugs: skipfish and sqlmap does not detect
sql-inj with output in HTTP header

• Fastest scanner: w3af
Thursday, May 24,

Contacts
• Karim: karim@lvk.cs.msu.su • Andrew: petand@lvk.cs.msu.su

Thursday, May 24,

Sign up to vote on this title
UsefulNot useful