Cybercrime

 Kill  Chain  vs.  

     Effec4veness  of  Defense  Layers  
Dr.  Stefan  Frei    &    Francisco  Artés  
@stefan_frei                  @franklyfranc  

Trusted Advice. Measured.

THE  FLIGHT  TO  ABU  DHABI  TOOK  
LONGER  THAN  TESTING  IPS.  
 Speaker  –  Dr.  Stefan  Frei  
§  Professional  
§  Research  Director  @  NSS  Labs  
§  Research  Analyst  Director  @  Secunia  
§  Senior  Researcher  &  Pentester  
 @  ISS  X-­‐Force  

§  Contact  
§  Email:  sfrei@nsslabs.com  
§  TwiKer:  @stefan_frei  
 Speaker  –  Mr.  Francisco  Artés  
§  Professional  
§  Research  Director  @  NSS  Labs  
§  CSO/CISO    
§  Trace3  
§  Deluxe  Entertainment  
§  Electronic  Arts  
§  Contact  
§  Email:  frank@nsslabs.com  
§  TwiKer:  @franklyfranc  
ABSTRACT  
Cybercriminals  persistently  challenge  the  security  of  organiza4ons  through  the  
rapid  implementa4on  of  diverse  aKack  methodologies,  state  of  the  art  
malware,  and  innova4ve  evasion  techniques.  In  response  organiza4ons  deploy  
and  rely  on  mul4ple  layers  of  diverse  security  technologies.  This  talk  examines  
the  aKackers'  kill  chain  and  the  measured  effec4veness  of  typical  defense  
technologies  such  as  Next  Genera4on  Firewalls,  Intrusion  Preven4on  Systems  
IPS,  An4virus/Malware  Detec4on,  and  browsers  internal  protec4on.  Empirical  
data  on  the  effec4veness  of  security  products  derived  from  NSS  Labs  harsh  real  
world  tes4ng  is  presented  together  with  a  live  demonstra4on  of  successful  
evasion  of  malware  detec4on.  We  find  a  considerable  gap  of  protec4on  levels  
within/and  across  different  security  product  groups.  Using  Maltego  complex  
correla4ons  between  undetected  exploits,  crimware  kits,  and  affected  so^ware  
vendor  and  products  are  demonstrated.  
Agenda  

§  How  we  get  aKacked  

§  Layered  Defense  
§  Results  from  NSS  Labs’  tes4ng  
§  Demonstra4on  of  Exploit  vs.  Layered  Defense  
§  Conclusion  
 
AKack  Kill  Chain  
–  AKacker  vs.  Defender  

AKackers  View  

Prepare(A:ack(( Detec.on( Target( Value(

Method/Tools( Evasion( Exploita.on( Extrac.on(

attack detection / prevention breach detection

Defenders  View  
AKack  Kill  Chain  
–  Understanding  the  AKacker  
Understand  the  threat  and  the  
⌃aKackers  mo4va4on  &  methods  

Prepare(A:ack(( Detec.on( Target( Value(

Method/Tools( Evasion( Exploita.on( Extrac.on(

attack detection / prevention breach detection

AKack  Kill  Chain  
–  Understanding  Evasion  
Understand  how  malware  
bypasses  detec4on  
⌃
Prepare(A:ack(( Detec.on( Target( Value(
Method/Tools( Evasion( Exploita.on( Extrac.on(

attack detection / prevention breach detection

⌃
Assess  the  effec4veness  
of  layered  defenses  
AKack  Kill  Chain  
–  If  preven4on  failed  

Prepare(A:ack(( Detec.on( Target( Value(

Method/Tools( Evasion( Exploita.on( Extrac.on(

attack detection / prevention breach detection

⌃
Detect  &  
neutralize  
 The  Changing  Threat  Environment  

Fastest  growing  
  segment  
Mo4va4on  

Personal   TheD    
Gain    
Author  
Tools  created  by  
Personal   of  
  experts  now  used  
Fame   Tools   by  less-­‐skilled  
Vandalism    
  criminals,  
Curiosity   for  personal  gain  

Script-­‐   Hobbyist   Expert  

Kiddy   Hacker  
AKackers’  Exper4se  
 Malware  Development  &  Tools  

§  Cybercriminals  developed  formidable  tools  

Easy  to  use  development  tools,  Q&A,  and  service  
level  agreements  just  as  in  every  mature  industry  
§  Detec4on  Evasion  and  Resilience  
By  design,  malware  is  developed  and  deployed  with  
detec4on  evasion  in  mind  
  Malware  Development  Process  

1  
Development   1.  Create  malicious  tool   1  x  

2.  Obfuscate  malware,  

2  
Evasion   create  permuta4ons   10,000  x  
3.  Test  against  detec4on  
3  
Q  &  A   engines   5,000  x  

4.  Deploy  undetected  samples  

4  
Deployment  
  Underground  Market  

Malware  offered  for  $249  

with  a  Service  Level  
Agreement  and  
replacement  warranty  if  the  
crea4on  is  detected  by  
any  anP-­‐virus  within  9  
months  
  The  Availability  of  Malware  Tools  

Results  in  a  high  degree  of  aTack  automaPon  

from  systema4c  iden4fica4on  of    
targets  to  fully  automated  exploita4on  
 

Leads  to  an  increase  in  opportunisPc  aTacks  

as  the  a=acker  no  longer  needs  exper4se  or  
special  skills  

Any  enterprise  can  become  a  vic1m  of  a3ack:  

⌃

at  any  1me,  for  any  reason,  and  without  being  

specifically  targeted.  
Automated  vulnerability  scanners  
and  aKack  tools  cannot  
differen4ate  if  you  consider  
yourself  a  high-­‐risk  target  or  not.    
 Our  Response:  Layered  Security  
We  respond  and  rely  on  layered  security  
Key  Security  Technologies  available:  
§  Network  Firewall    
§  Next  Genera4on  Firewall  
§  Intrusion  Preven4on  Systems  (IPS)  
§  An4virus  /  An4malware  
§  Browser  Protec4on  

How  effec1ve  is  the  defense  ?  

⌃

       How  do  we  know?  

  Layered  Defense  -­‐  Perimeter  

Perimeter
Firewall   Firewall  

IPS   IPS  

server desktop laptop

on premise off premise
  Layered  Defense  –  Host  Based  

Perimeter
Firewall   Firewall  

IPS   IPS  

An4   An4  

Host based
Virus   Virus  

Browser   Browser  
URL  Block   URL  Block  

server desktop laptop

on premise
on premise off premise
off premise
  Layered  Defense  –  Direct  AKack  
direct attack

Perimeter
Firewall   Firewall  

IPS   IPS  

An4   An4  

Host based
Virus   Virus  

Browser   Browser  
URL  Block   URL  Block  

server desktop laptop

on premise
on premise off premise
off premise
  Layered  Defense  –  Indirect  AKack  
direct attack indirect attack indirect attack

Perimeter
Firewall   Firewall  

IPS   IPS  

An4   An4  

Host based
Virus   Virus  

Browser   Browser  
URL  Block   URL  Block  

server desktop laptop

on premise
on premise off premise
off premise
  Layered  Defense  –  Side  channel  AKack  
direct attack indirect attack indirect attack

Perimeter
Firewall   Firewall  

IPS   IPS  

An4   An4  

Host based
Virus   Virus  

Browser   Browser  
URL  Block   URL  Block  

sidechannel
attack
server desktop laptop
on premise off premise
Or  any  of  these:  
We  are  doing  this:  
Wizard-­‐like  knowledge…    
  Engineering  Workflow  ..  

..  sadly,  security  tes4ng  is  not  that  simple  

It’s  more  like  this  -­‐  
  Where  does  the  data  come  from?  

§  Mul4-­‐million  dollar  research  and  tes4ng  

facility  in  Aus4n,  Texas  
§  Capable  of  24  x  7  tes4ng  
§  Global  research  network  captures  Internet  
threats,  zero-­‐days  &  trends  live,  as  they  arise  
 Security  Test  Metrics  

To  determine  the  security  effec4veness  

of  devices,  the  following  metrics  were    
used:  
 

1.   Exploit  Block  Performance  

2.   An4  Evasion  Performance  
3.   Performance  &  Leakage  
4.   Stability  &  Reliability  
 Metric  
Exploit  Block  Performance   1  

§  The  same  types  of  aKack  as  used  by  modern  cyber  
criminals  
§  U4lizing  mul4ple  commercial,  open  source  and  
proprietary  tools  as  appropriate  
§  More  than  1,400  exploits,  tested  such  that  
§ a  reverse  shell  is  returned,  allowing  the  aKacker  to  
execute  arbitrary  commands  
§ a  malicious  payload  is  installed  
§ a  system  is  rendered  unresponsive  
 Metric  
An4  Evasion  Performance   2  

§  Providing  exploit  protec4on  without  factoring  in  

evasion/obfusca4on  is  misleading  
§  Addi4onal  test  cases  are  generated  for  each  
appropriate  evasion  technique.    
•  At  TCP,  IP,  and  applica4on  protocol  level  
•  Fragmenta4on,  Segmenta4on,    
Obfusca4on,  Encoding,  Compression    
and  all  combina4ons  thereof  
 Metric  
Performance  and  Leakage   3  

§  Trade-­‐off  between  security  effec4veness  and  

performance  
Ensure  vendors  don’t  take  security  shortcuts  to  maintain  or  
improve  performance    
§  Evaluated  based  upon  three  traffic  types  
Based  on  hundreds  of  metrics  such  as  connec4on  rates,  latency,  
delta  in  performance  with  different  packet  sizes  and  HTTP  
response  sizes,  stateful/connec4on  tracking  capabili4es,  ..  
§  a  mix  of  perimeter  traffic  common  in  enterprises  
§  a  mix  of  internal  traffic  common  
 in  enterprises  
§  21KB  HTTP  response  traffic  
 Metric  
Stability  &  Reliability   4  

§  Long-­‐term  stability  is  par4cularly  important  for  

an  in-­‐line  device  
Verify  the  stability  of  the  device  under  test  
§  Tests  the  ability  to  maintain  security  
effec4veness  under  normal  &  malicious  traffic  
load  
Products  that  are  not  able  to  sustain  legi4mate  traffic  
(or  which  crash)  while  under  hos4le  aKack  will  not  pass  
Security  Effec4veness  

§  Security  Effec4veness  

combines  measured  cost  of  ownership,  security  
protec4on,    performance,  leakage,  and  stability    
§  Security  Value  Map  (SVM)  
shows  security  effec4veness  and  value  (cost  per  
protected  Mbps)  of  tested  product  configura4ons  
§  Customizable  
SVM  is  customizable  to  reflect  individual  weights  of  the  
different  factors  
 NSS  Labs  tested:  
6   Network  Firewalls  
Q3/2012  

15   Intrusion   Preven4on   S ystems  

Q3/2012  

13   End-­‐point  An4virus  Suites  

Q4/2012  
Browsers  
4   Q3/2012  

6   Next  Genera4on  Firewalls  

Q4/2012  
 
 
¤    Network  Firewalls  

§  Three  of  the  six  products  tested  crashed  when  subjected  
to  our  stability  tests  
This  lack  of  resilience  is  alarming  and  indicates  the  presence  of  a  vulnerability  
that  could  be  exploited  

§  Performance  claims  in  vendor  datasheets  are  generally  

grossly  overstated  
Performance  based  on  RFC-­‐2544  (UDP)  does  not  reflect  real  world  
environments  
§  Five  of  the  six  products  failed  the  TCP  Split  Handshake  
test    
Allowing  an  aKacker  to  reverse  the  flow  and  bypass  security.  Four  vendors  
released  a  patch  within  a  month  
¤    Network  Firewalls  

§  Longstanding,  tried,  and  field  proven  technology,  such  as  

firewalls,  can  s4ll  fail  on  basic  networking  aKacks  
§  AKacks  never  expire  –  security  devices  must  maintain  
protec4on  for  the  complete  range  of  aKacks  
§  Independent  tests  are  valuable  to  iden4fy,  and  have  
vendors  remediate  shortcomings    
¤    Intrusion  Preven4on  Systems  IPS  

400"

350"
§  Exploit  block  rate  varies  
300"
Undetected  Exploits   between  77%  and  98%  
(0f  1,486  tested)    

250"
§  Tuning  of  the  IPS  policy  makes  
200"
a  difference,  up  to  50%  less  
150"
Mean"74"exploits" protec4on  with  default  policy  
100"

50" §  Evasion  detec4on  has  

0" improved  considerably,  all  but  
Juniper"SRX"3600"

Tipping"Point"
PaloAlto"PA"5020"
SonicWall"

McAfee"M80000"
ForFGate"3240C"
StonesoI"1302"
CheckPoint"12600"

Sourcefire"Virtual"
Sourcefire"3D8260"
Sourcefire"8120"
Sourcefire"8250"
IBM"GX"7800"

McAfee"M8000"
Juniper"IDP"8200"

one  vendor  tested  passed  

 ¤    Intrusion  Preven4on  Systems  IPS  

800$
714$ §  Correla4on  of  undetected  
700$
exploits  between  vendors  
600$ Unique  Exploits  undetected   products  
by  N  Vendors  IPS  
Number$of$Exploits$

500$
§  Only  a  small  set  of  exploits  is  
400$
Three$exploits$that$
required  to  successfully  bypass  
300$ 244$ are$undetected$by$ all  IPS  products  
7$of$10$vendors$IPSs$
200$
§  Only  one  combina4on  of  
89$
100$ 52$
29$
different  IPS  products  blocked  
11$ 3$ 0$ 0$ 0$
0$ all  exploits  
1$ 2$ 3$ 4$ 5$ 6$ 7$ 8$ 9$ 10$
Number$of$IPS$vendors$
¤    End-­‐Point  An4virus  

Percent  undetected  exploits   §  AV  products  differ  up  to  58%  in  
Kaspersky#
(of  144  exploits  tested)   block  performance  
Avast#
Norton# §  Many  products  failed  to  detect  
AVG# exploits  over  HTTPS  that  were  
detected  over  HTTP  
ESET#
Trend#Micro#
McAfee#
§  Keeping  AV  up-­‐to-­‐date  does  
Avira#
MicrosoC# not  yield  adequate  protec4on,  
F=Secure# s4ll  many  old  exploits  remain  
undetected  
Norman#
Panda#
Total#Defense#

0%# 10%# 20%# 30%# 40%# 50%# 60%# 70%# 80%# 90%# 100%#
¤    Browser  Block  Performance  

§  Browsers  offer  the  largest  aKack  surface  in  most  enterprise  
networks  
§  Browsers  are  the  most  common  vector  for  malware  
installa4ons  
 

§  NSS  Labs  con4nuously  measures  browsers  block  performance  

since  2011   Software Stacks
 

VM1   VM2   VM3   VM4  

URL Feeds
¤    Browser  Block  Performance  

Suspicious  URL  block  performance  

¤    Browser  Block  Performance  

§  Internet  Explorer  maintained  a  malware  

block  rate  of  95%  
§  Firefox  and  Safari’s  block  rate  was  just  under  6%  
§  Chrome’s  block  rate  varied  from  13%  to  74%  
Percent$blocked$URLs$

Safari$ 5%$

Firefox$ 5%$

Chrome$ 28%$

Internet$Explorer$ 94%$

0%$ 20%$ 40%$ 60%$ 80%$ 100%$

Opportunity  for  Cybercriminals  

#  
=   #   exploit  
exploits  
x              targets  
                               x       availability  
  Undetected  Exploits  
undetected Exploits  that  bypass  
exploits our  defense  layers  
(IPS,  NGFW,  
An4virus,  ..)  

Sadly  enough,  these  exploits  exist  and  are  

plen4ful  ..  
  Exploits  for  prevalent  programs  
undetected Exploits  that  bypass  
exploits our  defense  layers  
(IPS,  NGFW,  
An4virus,  ..)  

Exploits  that  hit  

popular  programs  
with  large  market  
share  
prevalent &
vulnerable programs
Exploits  for  popular  programs  are  a  dangerous  
beast  ..  
  Proven  and  readily  available  exploits  
undetected Exploits  that  bypass  
exploits our  defense  layers  
(IPS,  NGFW,  
An4virus,  ..)  

Exploits  that  are  

Exploits  that  hit   readily  available  in  
popular  programs   crimeware  kits  or  
with  large  market   penetra4on  tes4ng  
share   tools  
prevalent & exploits available
vulnerable programs in crimeware kits
Make  them  readily  available  for  everyone  with  a  
criminal  mid  calls  for  disaster!  
  Failure  of  the  security  industry  
undetected
exploits

prevalent & exploits available

vulnerable programs in crimeware kits
Security  products  failing  to  detect  these    
exploits  are  hardly  acceptable  
Demonstra4on  
  Undetected  Exploits  vs.  Metasploit  
Correla4on  of  exploits  not  detected  by  IPS/NGFW  with  exploits  available  in  Metasploit  
Many  publicly  available  and  easy  to  use  exploits  bypass  detec4on  

Undetected  exploits  
available  in  Metasploit  

26%  of  866  Metasploit  

Undetected   exploits  are  not  detected  
exploits   by  at  least  one  IPS/NGFW  
  Correla4on  of  undetected  Exploits  
Exploits  available  in  crimeware  kits  are  s4ll  undetected  by  IPS  or  NGFW  engines.  
43  of  117  exploits  that  could  be  aKributed  to  crimeware  kits    bypassed  detec4on    
of  9  of  23  detec4on  engines  

Phoenix  

Eleonore  

IPS/NGFW  devices   Undetected  exploits  

Crimeware  kits  
that  missed  exploits   from  crimeware  kits  
  Undetected  Exploits  vs.  AKacked  Vendor  
Correla4on  of  exploits  not  detected  by  IPS  or  NGFW  with  the  so^ware  vendors  of  the  
programs  targeted  by  these  exploits  
Most  undetected  exploits  target  Microso^  products  –  relevant  exploits  go  undetected!  

Exploits  against   Microso^  

Microso^  products  
  Correla4on  of  undetected  Exploits  
Many  exploits  are  not  detected  by  several  IPS  engines  
714  of  1,486  exploits  tested  are  not  detected  by  at  least  one  IPS  engine,    
40%  or  286  by  at  least  two  IPS  engines    

Undetected  by  
one  IPS  

Undetected  by  
mul4ple  IPS  
Bubble  size  
indicates  number    
of  IPS  engines  not    
detec4ng  given  exploit  
Combined  Failure  Rate  

Attacker Layered Defense Target

Device  A   Device  B  

Failure Rate Failure Rate Combined Failure Rate

PA PB
PA B ¢

10% 10% ?

PA B = PA . PB = 1%
¢ (?)
Correla4on  Fallacy  
-­‐  Rethink  your  risk  assessment  

PA B ≠ PA PB
¢
.
§  Failures  are  correlated,  they  are  not  
independent  events  

§  The  combined  failure  rate  

is  typically  considerably  higher   P
¢A B > PA PB

Conclusion  &  Findings  
 
§  Vendor  claims  on  the  effec4veness  or  
performance  of  products  are  frequently  
overstated,  or  based  on  non-­‐realis4c  
assump4ons  
§  Several  network  firewall  products  tested  crashed  
when  subjected  to  our  stability  tests  
§  An4virus  does  not  prevent  a  dedicated  aKacker  
from  compromising  a  target  
§  Several  products  failed  detec4on  of  exploits  
when  switching  from  HTTP  to  HTTPS    
Recommenda4ons  

§  There  is  no  product  or  combina4on  of  products  

tested  by  NSS  Labs  that  provide  100%  protec4on  
§  Assume  that  you  are  already  compromised  
§  Organiza4ons  should  complement  preven4on  
with  breach  detec4on  and  SIEM  to  iden4fy  and  
act  on  successful  security  breaches  in  a  4mely  
manner  
§  Access  to  independent  informa4on  on  security  
product  effec4veness  and  performance  is  
important    
Complexity  

§  Technology  alone  cannot  provide  the  highest  

protec4on  

§  Competent  and  mo4vated  security  personal  is  

key  to  effec4ve  security  –  and  make  the  best  
use  of  the  tools  
Trusted Advice. Measured.

Thank  you  
 
sfrei@nsslabs.com  
frank@nsslabs.com  
Resources  
§  Network  Firewall  Group  Test  2011  
hKps://www.nsslabs.com/reports/network-­‐firewall-­‐group-­‐test-­‐2011  
or  hKp://bit.ly/RzLX3a  

§  IPS  Compara4ve  Analysis  2012  

hKps://www.nsslabs.com/reports/ips-­‐compara4ve-­‐analysis-­‐2012  
or  hKp://bit.ly/SvHwQ  
 

§  Consumer  AV/EPP  Compara4ve  Analysis  -­‐  Exploit  Protec4on  

hKps://www.nsslabs.com/reports/consumer-­‐avepp-­‐compara4ve-­‐analysis-­‐exploit-­‐protec4on  
or  hKp://bit.ly/S5Mqs7  

§  Is  Your  Browser  Puyng  You  At  Risk?  

hKps://www.nsslabs.com/reports/your-­‐browser-­‐puyng-­‐you-­‐risk-­‐part-­‐1-­‐general-­‐malware-­‐blocking  
or  hKp://bit.ly/SvGHur  

§  Targeted  Persistent  AKack  (TPA)  

hKps://www.nsslabs.com/reports/analysis-­‐brief-­‐targeted-­‐persistent-­‐aKack-­‐tpa-­‐misunderstood-­‐
security-­‐threat-­‐every-­‐enterprise  
or  hKp://bit.ly/SvGO99