Asmv12ddosbestpracticev8 Final 160711160608 PDF

BIG IP ASM V12
DDOS PROFILE
Global Service Tech Summit, Seattle
Sep, 2015, v3
Lior Rotkovitch, NPI

ASM, L7 DDoS & Analytics
lior@f5.com
ASM DDoS Profile
DDOS - HTTP FLOODS ATTACKS DDOS - BOTS
From single IP to single URL Simple bots

From multiple IPs to single fixed URL Impersonating Bots
From multiple IPs to multiple fixed URLs Bots with cookies & JS capabilities
From multiple IPs to multiple random URLs Bots acting as full browser
From multiple IPs from a specific country
Fine Tune your Thresholds & Reporting Reporting
F5 Networks, Inc 2
HTTP Floods facts: Legitimate Layer 7 requests
Asking a web page thousands of time instead of one (few) times
Exhausting backend servers resources: memory, CPU, Disk etc

Users Or Bots
Relatively easy to execute with simple tools
Not easy to detect the offending source nor to prevent it
Wrong identification will prevent valid users from accessing the site (false positive )
User
Source IPs Unidentified User Web Site
Hacktivism
Servers Database
Google Web Bot

Web Bot
F5 Networks, Inc 3
HTTP Floods types Requests increase from or/and to URLs inside the web site
From single IP to single URL

From multiple IPs to single fixed URL
Users Or Bots
From multiple IPs to multiple fixed URLs
From multiple IPs to multiple random URLs
From multiple IPs originating from a specific country
User
Hacktivism
Servers Database
Google Web Bot

Web Bot
F5 Networks, Inc 4
ASM Detection & Mitigation concept - HTTP Floods
ASM process:
Users Or Bots 1. Monitoring entities: RPS Latency IPs URLs
2. Detecting Increase
3. Activating Mitigation
Source IPs
User
RPS
Latency App URLs &
objects
Hacktivism
Servers Database
Google Web Bot

Web Bot
F5 Networks, Inc 5
ASM Detection & Mitigation concept DoS Profile
Location: Security DoS Protection DoS Profiles dos
F5 Networks, Inc 6
TPS Based Detection: Transaction Per Second based detection
and mitigation
Client: Hey server, can I get the web page ?
Server
ASM: no, you are sending too many requests.
You will have to

Answer CSID
Answer CAPTCHA
Be Rate Limited / Blocked
F5 Networks, Inc 7
TPS Based Detection
Monitoring Request Per Second increase form source IP, Geo, URL, Site Wide.
Then apply one of the mitigation policies: CSID, CAPTCHA, Rate limit
F5 Networks, Inc 8
1
TPS Based Detection
1. BY Source IP (Detection & Mitigation Polices) a
2b
2. Mitigation polices: c
3
a) Client Side Integrity Defense
b) CAPTCHA challenge
c) Request Blocking
4
3. By Geolocation (Detection & Mitigation Polices)
4. By URL (Detection & Mitigation Polices)

5
5. By Site Wide (Detection & Mitigation Polices)
6. Prevention Duration
6
F5 Networks, Inc 9
By Source IP: Detection Criteria
Detection: thresholds for determining DDoS attack - by source IP increase
Mitigation: which mitigation will apply on the offending source IP
Detection
Mitigation
F5 Networks, Inc 10
Detection Ratio
Ratio thresholds - measuring ratio with two time interval:
Long (History Interval): Measure the last 1 hour RPS averages every 10 seconds
Short (Detection Interval): Measure the last 10 seconds RPS averages every 10 seconds
F5 Networks, Inc 11
Detection Ratio
Example:
Long (History Interval): 50 TPS TPS increased by: ((370 - 50) /50)*100 = 640%
Short (Detection Interval): 370 TPS 640% > 500% = True
F5 Networks, Inc 12
At least X Transactions:
Detection Ratio A minimum condition to
prevent false positive
increase (source IP
starts browsing the site
and goes from 0 to 30
RPS)
TPS increased by % AND minimum fix number of transactions
Example: 640% AND 40 = True
F5 Networks, Inc 13
Detection Ratio TPS reached:

or
Fixed Ratio thresholds
ORed with fixed TPS
(TPS increased by % AND minimum fix number of transactions) OR TPS reached
Example: 640% AND 40 OR 200 = True
F5 Networks, Inc 14
1
TPS Based Detection
2b
c
2. Mitigation polices:
3
c) Request Blocking 4

5
6
F5 Networks, Inc 15
Client Side Integrity Defense Concept
Server
ASM: no, you are sending too many

if a browser:
requests. Are you a browser ?
Yes, Im a browser
User
ASM: ok, you are allowed. Here is the web
If a bot: page you asked for.
*^lkjdfg@#$
Web Bot
ASM: Bye Bye Blocked
F5 Networks, Inc 16
Client Side Integrity Defense Concept
Checking JavaScript capabilities
A client is considered legitimate if it meets the following criteria:
The client support JavaScript
The client support HTTP cookies
The client should calculate a challenge inside the JS
If satisfied = legitimate client that can access the site
F5 Networks, Inc 17
Client Side Integrity Defense - Flow
User Browser DoS Profile App
First main page access
HTTP Request (no cookie)
Send JS test
Computational challenge
This is the flow and timeline
Solve challenge/
set cookie with time stamp of events.
Transparent to the user,
HTTP Request (cookie) Reconstruct request
done under the hood
Original HTTP Request
Note that request is held at
HTTP Response (main page)
the ASM and not arriving the
app until checks are
More object requests (cookie)
satisfied
Validate cookie: format & time stamp
Not all checks are described
More object requests
here, some are internal IP.
More responses
Deliver page More responses
F5 Networks, Inc 18
Client Side Integrity Defense JavaScript sample
The JS is obfuscated
From user perceptive this
is transparent action.
F5 Networks, Inc 19
Client Side Integrity Defense Mitigation summary
If no reply No problem for us
If didnt solve the challenge but still sending request Block (RST)
If did solve the challenge but:
Cookie is wrong format Block (RST)
Time stamp expired Block (RST)
If client access a resource (image) without getting the cookie first Block (RST)
F5 Networks, Inc 20
1
TPS Based Detection
2b
3
c) Request Blocking
4

5
6
F5 Networks, Inc 21
CAPTCHA Challenge - Concept
Server
ASM: no, you are sending too many

If a user: requests. Please answer this CAPTCHA
OK, I answered challenge, show me your human !:
User ASM: ok, you are allowed. Here is the web

page you asked for.
If none user:
Ha ? *^lkjdfg@#$
ASM: Bye Bye Block him dood !
Web Bot
F5 Networks, Inc 22
CAPTCHA Challenge Completely Automated Public Turing test to tell
Computers and Humans Apart
Ultimate solution for identifying human or bot

Send challenge to every IP that reached IP detection criteria thresholds
To CAPTCHA or not to CAPTCHA ?

Some argues that CAPTCHA is not a good usability because an innocent user gets
CAPTCHA and he will not know why. So, remember that a valid user should pass
browser tests. i.e. if a user is blocked (or gets a CAPTCHA) there is a reason and
maybe he is not innocent (infected ?) .
F5 Networks, Inc 23
CAPTCHA customize response
Can be customize to the web site look and feel colors via css
Failure Response page is served if the first attempted fails
F5 Networks, Inc 24
CAPTCHA Challenge - Flow
Request mypage.php
GET /mypage.php (no cookie)
Send CAPTCHA
While the system is still in a
CAPTCHA HTML +JS response state of attack the offending
CAPTCHA rendered Cookie with time stamp source will be presented with
another CAPTCHA every 5 min.
Solve CAPTCHA
Same as CSID, request is held
Submit CAPTCHA
solution
at the ASM until CAPTCHA is
solved
GET /mypage.php + CAPTCHA
cookie
Verify CAPTCHA solution
Validate cookie
GET /mypage.php
mypage.php
HTML of mypage.php HTML of mypage.php
rendered
F5 Networks, Inc 25
CAPTCHA mitigation summary
If didnt submit the challenge - no request DOSing us
If didnt solve the challenge but still sending us attacks Blocked
If did solve the challenge but:
Cookie is wrong format RST
Time stamp expired 5 min RST
F5 Networks, Inc 26
1
TPS Based Detection
2b
3
c) Request Blocking
4

5
6
F5 Networks, Inc 27
Request Blocking / Rate limit
ASM: no, you are sending too many requests.

Im limiting your requests sending rate
While CSID and CAPTCHA try to understand who is the offending source (bots or browsers)
request limiting is indifferent to the identity and limits the offending sources.
F5 Networks, Inc 28
Request Blocking
Request Blocking:
Blocking: block all IPs from the offending source if a source IP reached thresholds I dont
want him on my site at this point
Rate Limit: limit the amount of allowed request from the offending source if reached
thresholds I can sustain only some of the traffic at this point
F5 Networks, Inc 29
Request Blocking Mitigation Summary
Block all blocking all traffic from the offending

source (i.e. I dont want to see any more traffic from
this source)
Rate Limit rate limit the offending source
Rate limit will limit to long (history) TPS rate
Example
If long was 50 TPS And increase in short is 150 TPS
Rate limit to 50 TPS
F5 Networks, Inc 30
TPS based: by source IP Summary
Measuring source IP increase
All source IPs that reached the thresholds will be presented with the enabled mitigation
If still increasing , fall back according to the order in the GUI (switching mitigation)
Client Side Integrity Check

CAPTCHA Challenge
Request Blocking
F5 Networks, Inc 31
1
TPS Based Detection
2b
c
2. Mitigation polices: 3

5
6
F5 Networks, Inc 32
HTTP Floods Geolocation detection and Mitigation
Users Or Bots
http floods type: From multiple source
IPs originating from a specific country
User
Hacktivism
Servers Database
Google Web Bot

Web Bot
F5 Networks, Inc 33
Geolocation - Detection
Geolocation Relative to the whole traffic of the site:
500 % request increase of the whole site from a specific country
AND
At least 10 % of the whole site traffic
F5 Networks, Inc 34
Geolocation Mitigation
All clients requests arriving from the specific country will be presented with mitigation:

CAPTCHA Challenge
Request Blocking
(note that blocking will block

all users from this country)
F5 Networks, Inc 35
Geolocation Black n White listing
Specifies the countries that the system always blocks

whenever the system is in a state of DDoS detection.
Done regardless of the thresholds set in the DDoS
profile
Allows access to the web site regardless of
geolocation detection criteria thresholds only
i.e. other thresholds still apply
F5 Networks, Inc 36
1
TPS Based Detection
2b
3
c) Request Blocking
4

5
6
F5 Networks, Inc 37
HTTP Floods URL Detection and Mitigation
Measuring requests increase on a URL
Users Or Bots
Floods types:
From multiple IPs to multiple fixed URLs
Source IPs
User
Source IPs Unidentified User RPS Web Site
App URLs &

objects
Hacktivism
Servers Database
http://site.com/sell.php
Google Web Bot

Web Bot
F5 Networks, Inc 38
URL Detection Criteria
Collecting RPS on URLs
TPS increase by* AND at least X TPS **

Calculation:
*Ration of long and short
OR TPS reached
**Minimum TPS thresholds for detection
F5 Networks, Inc 39
URL Detection Criteria Mitigation
All clients that access the URL: Client Side Integrity Check
CAPTCHA Challenge
Request Blocking Rate limit (No block all)
F5 Networks, Inc 40
1
TPS Based Detection
2b
c
2. Mitigation polices: 3

5
6
F5 Networks, Inc 41
HTTP Floods Site Wide Detection and Mitigation
Users Or Bots Floods types:

Source IPs Cases where DDoS attack is under the radar
User
Source IPs Unidentified User RPS Web Site
App URLs &

objects
Hacktivism
Database
Servers
Monitoring: all entities

Google Web Bot
Web Bot
F5 Networks, Inc 42
Site-Wide Detection Criteria
Collecting RPS on the entire website (all entities URLs, IPs)
In some cases the floods will avoid thresholds for IP based or URL based.
Site wide provide another layer of detection and prevention
Detection: Ratio
Fixed
*TPS increase by AND Minimum TPS thresholds for detection

OR TPS reached
F5 Networks, Inc 43
Site-Wide Detection Criteria Mitigation
Prevention polices
All clients that access the site: Client Side Integrity Check
CAPTCHA Challenge
Request Blocking - only rate limit no blocking
F5 Networks, Inc 44
1
TPS Based Detection
a
2b
1. BY Source IP (Detection & Mitigation Polices) c
3
2. Mitigation polices:
c) Request Blocking
4
5
6
F5 Networks, Inc 45
Prevention duration
Escalate top down every 120 second if thresholds are still increasing

CAPTCHA Challenge De escalate - start from the top
Every 7200 seconds
Request Blocking
F5 Networks, Inc 46
Stress Based detection
Predictive Latency predict how long it will take to

serve a new incoming request
ASM: Hey server, how many more requests

can you handle ?
Im the server
Server: Im fine, keep on sending them
F5 Networks, Inc 47
Stress Based Detection and prevention concept
Client: Hey server , can I get the web page ?
Im the server
ASM: mmm let me check. The Server can take

.
.
additional incoming requests. you are allowed
After a while
Client: Hey server, can I get web pages again now ?
ASM: no, my backend latency is now too high
and you are sending too many request.
You will have to:
Answer CSID or
Answer CAPTCHA or
Be rate limit
F5 Networks, Inc 48
Stress Based GUI
Same concept as TPS

based: source IP, Geo, URL,
Site wide and their
mitigation policies.
Addition condition of
backend latency. i.e. only
when the two conditions
reach thresholds, then apply
mitigation policy.
Note: Can work together (operate in

parallel) with TPS based and act as
layers of protection (e.g. TPS based
does only CSID in alert mode and
Stress based does request blocking in
case of latency increase)
F5 Networks, Inc 49
Stress Based Detection & Mitigation
Similar to TPS based,

Quiz yourself, what each item
means ?
1. By Source IP
a) CSID
b) CAPTCHA
c) Request Blocking
2. By Geolocation
3. By URL
4. Site Wide
F5 Networks, Inc 50
Stress Based Detection thresholds condition
Latency TPS
threshold AND threshold
exceeded? exceeded? Mitigation Is activated when
two types of thresholds are
reached :
Latency thresholds
AND
Then: TPS thresholds
Activate
Mitigation Policy
F5 Networks, Inc 51
Stress Based Detection thresholds condition
TPS thresholds
stress detection
prevention
Example:
in order to apply a prevention policy, both TPS and
Automatic stress detection enters a
Latency thresholds must be exceeded, then the enabled
state of exceeding thresholds. This by
prevention policy is activated.
itself will not active the prevention.
Latency thresholds are not visible in the GUI, they are part Only when the TPS thresholds will
of automatic detection. exceed then the prevention policy is
activated.
F5 Networks, Inc 52
TPS based VS Stress based
Quick way to protect against DDoS. Im in trouble Allows the option to activate the Mitigation only
and I want to block now ! when the backed experiencing latency AND RPS
increase (I only want to block when the attack is
Fixed number on the TPS reached is very easy causing backend latency)
and useful. Also easy to detect offending sources
Provide Layers of defense and notify about backend
issues (not just DDoS)
Conclusion:
TPS based is quick while latency based
allows more granular approach
F5 Networks, Inc 53
Heavy URLs
Not all URLs are equal

Some are more attractive than others
F5 Networks, Inc 54
Heavy URLs
Heavy URLs are URLs that consume more processing resources
from the server
Are good application DoS point - Even few requests can DoS the app
Typical heavy URLs are search box, product IDs
http://site.com/serach.php?q=a
Heavy URL
Servers Database
Ho wow, this will take a while

Searching
..
hold on
..
Almost there.
F5 Networks, Inc 55
Heavy URLs concept
Automatically measures latency on URLs for 48 hours

and decide who is heavy
When any URL based mitigation is active, the heavy URLs

that was detected as heavy will also get the active
mitigation
F5 Networks, Inc 56
Heavy URLs concept
Heavy URL is another detection capability. Once it is reached the thresholds AND one of the By URL detection
thresholds are reached Then the URLs that consider heavy URLs will be applied with the active mitigation policy
F5 Networks, Inc 57
Heavy URLs configuration
Example: By URL TPS reached 1000 TPS and is currently applying CSID mitigation.
Heavy URL is enabled
F5 Networks, Inc 58
Heavy URLs Configuration
1. Automatic Detection - Automatically add URL

1 that will be considered as heavy
2. Manual Heavy URLs Manually add URL that

2 will be considered as heavy
3. Ignored URL Exclude those URL from

heaviness
4. Latency Threshold Above this threshold ->

3 heavy URL
F5 Networks, Inc 59
Heavy URL Reporting Security Reporting DoS Application URL Latencies
Example:
If search.php is defined as heavy and if index URL is currently being mitigated with CSID because it exceeded the
thresholds of URL reached (or fixed) then every source IP that is accessing search.php will also get the CSID check.
F5 Networks, Inc 60
Remember, security is a process
Reporting first
Fine Tune your thresholds
Before DDoS Attack
During DDoS Attack
F5 Networks, Inc 61
First rule of detection - AVR Reporting
Know your web site metrics
AVR graphs help you understand the site metrics: Sources

IPs
URLs
Statistics Analytics Overview Site Wide
Statistics Analytics Transactions View by Geolocation
RPS
TPS
AVR graphs inside ASM tab: Latency
L7 DDoS measurements
Security Reporting DoS Overview
Security Reporting DoS Application Transaction outcome
F5 Networks, Inc 62
Why Fine Tune Thresholds ?
Out of the box thresholds are good for most web sites
Depending on the web site traffic fine tuning thresholds might be needed.
Fine tuning thresholds can be divided into:
Before DDoS Attack
During DDoS Attack
Good for me ??
F5 Networks, Inc 63
Fine Tune Thresholds Before attack
Process:
Pre requisite: Enable DDoS Profile on the desired virtual
1) White list IPs, geolocations countries, URLs (admin) etc
2) Get visibility with transparent mode write down metrics*
3) Test and decide which prevention will apply thresholds exceeds (TPS bases/
Latency based , heavy URL config etc)
4) Fine tune thresholds for fixed and ratio based
5) Switch to blocking When needed
*good list for L7 DDoS metrics

F5 Networks, Inc 64
Fine Tune Thresholds Before DDoS for Source IP
Go to Statistics Analytics HTTP Transaction

View by: Client IP address
List top TPS Avg IPs
F5 Networks, Inc 65
Fine Tune Thresholds Before DDoS for Source IP
By examining the client IP Address you can

conclude the averages of normal traffic you
expect to see from the top source IPs.
Knowing normal averages can help defining

the TPS increase by ratio.
The idea is that you can determine how much

traffic is allowed until assumed a ddos attack.
F5 Networks, Inc 66
Fine Tune Thresholds Before DDoS for Geolocation
Go to Security Reporting dos Application transaction outcome
The same concept works for the

geolocation thresholds graph.
From the drilldown choose Countries on

AVR reports
Which countries you expect to see traffic ?
F5 Networks, Inc 67
Fine Tune Thresholds Before DDoS for URL
The same idea applies to URLs.

Sort graph by URLs
Which URL should have to highest RPS ?
F5 Networks, Inc 68
Fine Tune Thresholds Before DDoS for URL
F5 Networks, Inc 69
Fine Tune Thresholds Before DDoS for Site Wide
On the drilldown choose

Virtual Server
This will help us understand

the over all traffic load that we
have when there is no DDoS
attack.
F5 Networks, Inc 70
Fine Tune Thresholds Before DDoS for Site Wide
Site wide = Virtual server
The overall traffic should be much higher than the other thresholds.
The values reflect the total amount of TPS that the virtual can handle.
F5 Networks, Inc 71
Fine Tune Thresholds During attack
Process:
1) Fine tune white list source if needed
2) Identify sources that exceed thresholds (source IPs, URLs, Geo, SiteWide) by
looking at reporting.
3) Determine the attack type: from fixed/random source IP to fixed/ random URL.
Conclude which of the detection types you need (source IP only ? Source IP and URL
based only ? etc. )
4) Fine tune thresholds according to the exceeding sources (ratio / fixed)
5) Apply mitigation and decide what is working and what is not. Uncheck the
mitigations that are not effective
6) Go to step 1 and repeat
F5 Networks, Inc 72
Fine Tune Thresholds During attack Source IP
Security Reporting dos

Application transaction
outcome

Client IP Address
F5 Networks, Inc 73
Fine Tune Thresholds During attack Geolocation

outcome

Countries
F5 Networks, Inc 74
Fine Tune Thresholds During attack URLs

outcome
On the drilldown choose URLs
F5 Networks, Inc 75
Fine Tune Thresholds During attack - Site Wide

outcome

Virtual Servers
F5 Networks, Inc 76
AVR reports and graphs
Security Event Logs DoS Application Events
Number
Host IP of TPS
Time line Mitigation type can help Attack ID : Clicking

Attack start / end
understand which of the will show graph
mitigation is effective and
when switching mitigation
occurred
F5 Networks, Inc 77
Security Reporting DoS Application Transaction Outcomes
High, medium and low

impact allow to filter
high impact attacks
and deal with them
first
impact is the latency

on the backend for all
entities. The higher
the latency the higher
the impact is.
F5 Networks, Inc 78
AVR reports and graphs Security Reporting DoS Application Transaction Outcomes
Start and End points - red flags indicate the start of an attack and the green flags indicate the end of
an attack. Switching mitigation can occure several time over the DDoS attack.
F5 Networks, Inc 79
Incomplete Indicates traffic that was dropped by the server because the connection was
incomplete or the server did not respond.
Blocked Indicates traffic that was blocked as a result of the mitigation policy (any of the
prevention policies including bots blocking)
Proactive Mitigation Indicates the amount of time that the proactive bot defense mechanism
was severed
CAPTCHA mitigation Indicates the amount of time that the CAPTCHA challenge was severed
to offending sources
CS integrity mitigation Indicates the amount of time that the client-side integrity defense
challenge was severed to offending sources
BIG IP Response Indicates traffic that is a response to the client from the BIG-IP system.
Cache by BIG IP Indicates traffic that is served from cache configured (WA, RamCache)
Whitelisted Indicates traffic from IP Address that are in the whitelist of the DoS profile
Pass through - Indicates traffic that is pass to the application trough ASM to the server
F5 Networks, Inc 80
The AVR DoS graph now
shows the thresholds
that are set in the TPS
detection tab.
The Display Thresholds

check box will display
them or clear them from
the graph.
F5 Networks, Inc 81
Fine Tune Thresholds Summary
Before DDoS:
Write the normal thresholds for the web site: (IPs, Geolocation, URLs, Site Wide)
Set the ratio and the fix threshold for each of the above detection criteria (How much the web
site can take 2 times the traffic , 5 times etc)
Test the configuration and the prevention policy, then conclude which one is good for you
During DDoS:
Identify the source IPs, URLs and entire site traffic increase and determine the attack type
Set the fixed TPS number in each of the above criteria and apply mitigation
Verify the results in the Transaction outcome graph
F5 Networks, Inc 82
DDoS Bots - Detection & Mitigation
F5 Networks, Inc 83
Layers of defense against Bots
Simple Bots
Gohogle
Impersonating Bots This bot section is mostly about
bots that DoS / DDoS. However,
Bots detection and prevention
Bots with cookies / JS capabilities can be used for various bot
problems the site is experiencing.
Bots acting as full browser
F5 Networks, Inc 84
DDoS Bots
Bots can be classify in many ways, mostly there are:
Users Or Bots
1. Simple bots
2. Impersonating Bots
3. Bots with cookies & JS capabilities
4. Bots acting as full browser
Google Web Bot
Web Site
Web Bot
Unidentified Servers Database

User
User
F5 Networks, Inc 85
Enabling Bot signatures protection
F5 Networks, Inc 86
Bots Simple Bot
Simple bot can be any command line tool such as:

curl , wget , ab
Im a simple Bot
Server
ASM: yes, I have your signature. Sorry

mate you are blocked.
F5 Networks, Inc 87
Categorizing Bots
Bad Bots aka Malicious

are well know command
line tools we want Bad Bots
them out
Good Bots aka Benign are

well know search engine and
monitoring tools we want
them in Good Bots
F5 Networks, Inc 88
Bot Signatures - Each categories include:
None ignore
Report report only used for monitoring
Block block
None
Report
Block
None
Report
Block
F5 Networks, Inc 89
Excluding specific bot signatures from category settings
Specific signature can be excluded from the

category setting
Search the signature in Available signature list
and move it to the left pane.
In this example ab tool will not be blocked
even if the category that includes it is in
blocking mode
F5 Networks, Inc 90
First - White list good Bots
3
DNS Server
Google
Im a google Bot Web Server
2
ASM: lets see if you really are. Im doing
Google Reverse DNS lookup.
4
Thanks Yes, I see that, please continue.
F5 Networks, Inc 91
White list good Bots -
with their domain name
DNS Server
3. The signature includes domain name.
ASM issue Reverse DNS query to verify
the origin of the request
1. Request arrive with User 4. Once approved ASM will allow the
Agent : Googlebot/2.1 2. ASM search the
google bot to access the web site
google bot signature
Web Server
User-Agent: Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)
F5 Networks, Inc 92
Bot Signature Repository
Security Options DoS Protection Bot Signatures List
Bot Signature repository for

the entire system is under
Options.
Bot signatures repository is

update with the ASM
signature update
F5 Networks, Inc 93
Bot Signature List: general signatures repository
Signatures can be sort by:

Signature Category
Signature Type:
Malicious / Benign
User Define signatures
Yes / No
Partition: signature can
be assign to a specific
partition
Clicking on any of the sorting

will change the order.
F5 Networks, Inc 94
Sorting the Bot Signature Repository
Various filtering
Create new Bot Signature
F5 Networks, Inc 95
Bot Signature Categories
Creating new category

for Malicious or Benign
F5 Networks, Inc 96
Create a new bot signature: simple edit mode
Bot Signature name
Domain name execute

reverser DNS query to
verify origin. Add the
domain if the Bot has one
Category
Simple edit mode: inside

a user agent header or
in a URL.
Create when done

F5 Networks, Inc 97
Create a new bot signature - advanced edit mode
Advanced Edit Mode - rule granularity

For full details consult F5 document
Signature syntax example:

headercontent: "sample_text"; useragentonly;
F5 Networks, Inc 98
Bot signature facts
Signatures associated with a domain name are

validated with reverse DNS lookup.
Blocking and reporting :
Block flag - resets the connection and reports the
action as "bot signature block" with the bot
signature name.
Report flag - Report bot name and categories
(AVR)
Updating bot signature as part of the ASM
signature update
F5 Networks, Inc 99
Bots Impersonating Bot
Gohogle DNS Server
Im a google Bot Web Server

ha ha ha
ASM: lets see if you are. Im doing Reverse

DNS lookup.
Gohogle
ASM: Hey DNS, whos this guy ?
DNS: no one important
Bummer ASM: you are not google bot

Bye Bye -> block this creature !
F5 Networks, Inc 100

Bots Impersonating Bot
DNS Server
3 The real google bot includes domain
name. ASM issue Reverse DNS query to
verify the origin of the request
1 Request arrive with User 4 If the source IP is not the expected one
Agent : Googlebot/2.1 2 ASM search the according to the DNS query ASM will block
google bot signature the impersonating bot
Web Server

Bots with cookies & JS capability
Im a bot that can Web Servers
understand JS and
support cookies
ASM: prove it, answer my challenges

Ha ?
No you are not, bye bye -> block this bot.
Bummer

Proactive Bot Defense
PBD is good for:

Bots that can handle JS
Bots that can handle JS and cookies
Bots floods
Under the radar bots
Block any bot accessing the site (humans only web site)

Proactive Bot Defense and Bot Signature
Proactive Bot
Defense is now
integrated with
the bot
signatures.
When enabling
proactive bot
defense the
bot signature
feature will be
enabled as
well

Proactive Bot Defense
Send Client side challenges to ALL client and thus mitigate bots all the time
Various challenges are sent and then validate by PBD blocked or allowed

PBD - Client side integrity defense - flow
First main page access
HTTP Request (no cookie)
Computational challenge
Solve challenge/
set cookie with time stamp
HTTP Request (cookie) Reconstruct request

Original HTTP Request
More object requests (cookie)
Validate cookie: format & time stamp

More object requests
More responses
Deliver page More responses

Proactive Bot Defense configuration
Always sending CS all the time
During attack only if other component of the dos profile is in dos mode PBD
will send the CS challenge (acting as two layers of mitigation)
This allows second layer of protection (rate limit and PBD)
Grace period - cookie expiration time 300 = 5min
White list exclude PBD on those IPs

Bots acting as full browser
DNS Server
Im a Bot that
Web Server
simulate browser
ASM: ok, what are your capability ? If you will not

answer right you will have to answer a CAPTCHA
Capability ?
CAPTCHA ?
You are not human, byyyye -> block this unhuman !
Bummer

PBD Additional bots identification with capabilities script
Bots: Bots acting as full browsers - Browser Simulation

How bots that simulate
browsers are evaluated ?
Block Suspicious Browsers addition tests are done to understand if this is a bot or a browser. ASM
will evaluate the source and will give it a score:
if the score indicates that the source is a bot it will block it.
If the score indicate uncertainty and if CAPTCHA challenge is checked, then CAPTCHA will be presented
to the source. If answer it is a human if not, blocked.

Block Suspicious Browsers
If Block Suspicious Browsers is unchecked send CS Challenge

If Block Suspicious Browsers is checked and CAPTCHA is checked send Client
Capabilities challenge and give it a score: If score in doubt send a CAPTCHA for
human verification
If Block Suspicious Browsers is checked but CAPTCHA Challenge is unchecked do
not send CAPTCHA and only block if the score is more than a human
Client Capabilities -challenge script flow
First request GET /sell.php
GET /sell.php (no cookie)
Client Capabilities Challenge response
Return Client Capabilities

verification 1. Authenticate and decrypted JS results,
2. Verify capabilities and set a score
3. Determine an action based on score
Blank page & Set cookie
Original HTTP Request + cookie Reconstruct request
HTTP Response (cookie)

HTTP Response
Validate cookie: format

GET /img.png (cookie)
& time stamp
GET /img.png (cookie)

DoS Bots Reporting

Bot signatures simulation Reporting dos Application Transaction outcomes
Transaction outcomes
is very useful for
monitoring traffic and
indicates various
measurements

Bot signatures simulation
Analytics HTTP throughput request throughput
AVR will provide details on DoS bot signatures (use drill downs )

Summary
Simple bots can easily be detected and blocked
White listing of bots = visibility to bot access and keep other bots out
Impersonating bots can be monitored / blocked
Bots that support JavaScript and cookies can now be noticed and be blocked
Reporting on the visiting bots to your web site is available via AVR
Custom bots signature is powerful tool to deal with bots
Bots signature is updating via the ASM signatures update

Resources
Our documentation is free for all. Read and learn more:

BIG-IP Application Security Manager Operations Guide
BIG-IP Application Security Manager: Getting Started
BIG-IP Application Security Manager: Implementations
BIG-IP Application Security Manager: Custom Signature Reference
BIG-IP Analytics: Implementations

Asmv12ddosbestpracticev8 Final 160711160608 PDF

Uploaded by

Document Information

Original Title

Copyright

Available Formats

Share this document

Share or Embed Document

Sharing Options

Did you find this document useful?

Is this content inappropriate?

Copyright:

Available Formats

Asmv12ddosbestpracticev8 Final 160711160608 PDF

Uploaded by

Copyright:

Available Formats

BIG IP ASM V12

Lior Rotkovitch, NPI

From single IP to single URL Simple bots

Asking a web page thousands of time instead of one (few) times

Exhausting backend servers resources: memory, CPU, Disk etc

Relatively easy to execute with simple tools

Not easy to detect the offending source nor to prevent it

Google Web Bot

From single IP to single URL

Google Web Bot

Google Web Bot

Client: Hey server, can I get the web page ?

ASM: no, you are sending too many requests.

You will have to

4. By URL (Detection & Mitigation Polices)

Ratio thresholds - measuring ratio with two time interval:

TPS increased by % AND minimum fix number of transactions

Example: 640% AND 40 = True

Detection Ratio TPS reached:

(TPS increased by % AND minimum fix number of transactions) OR TPS reached

Example: 640% AND 40 OR 200 = True

4. By URL (Detection & Mitigation Polices)

Client: Hey server, can I get the web page ?

ASM: no, you are sending too many

Checking JavaScript capabilities

A client is considered legitimate if it meets the following criteria:

The client support JavaScript

The client support HTTP cookies

The client should calculate a challenge inside the JS

If satisfied = legitimate client that can access the site

If no reply No problem for us

If did solve the challenge but:

Cookie is wrong format Block (RST)

Time stamp expired Block (RST)

4. By URL (Detection & Mitigation Polices)

Client: Hey server, can I get the web page ?

ASM: no, you are sending too many

User ASM: ok, you are allowed. Here is the web

Ultimate solution for identifying human or bot

To CAPTCHA or not to CAPTCHA ?

If didnt submit the challenge - no request DOSing us

If didnt solve the challenge but still sending us attacks Blocked

If did solve the challenge but:

Cookie is wrong format RST

Time stamp expired 5 min RST

4. By URL (Detection & Mitigation Polices)

Client: Hey server, can I get the web page ?

ASM: no, you are sending too many requests.

Block all blocking all traffic from the offending

Rate Limit rate limit the offending source

Rate limit will limit to long (history) TPS rate

Measuring source IP increase

Client Side Integrity Check

4. By URL (Detection & Mitigation Polices)

Google Web Bot

Geolocation Relative to the whole traffic of the site:

500 % request increase of the whole site from a specific country

At least 10 % of the whole site traffic

Client Side Integrity Check

(note that blocking will block

Specifies the countries that the system always blocks

4. By URL (Detection & Mitigation Polices)

App URLs &