1/36

DDoS protection using Netfilter/iptables

Jesper Dangaard Brouer
Senior Kernel Engineer, Red Hat
Network-Services-Team
Devon!"c# $e% &'()
Email* %rouer+red,at"com - netoptimi#er+%rouer"com - ,awk+kernel"org
DDoS protection
Using Netfilter/iptables
2/36
DDoS protection using Netfilter/iptables
.,o am / .,o am /

Name* Jesper Dangaard Brouer

0inu1 Kernel Developer at Red Hat

Edu* omputer Science !or 2ni" open,agen

$ocus on Network, Dist" s3s and 4S

0inu1 user since (556, pro!essional since (557

S3sadm, Kernel Developer, Em%edded

4penSource pro8ects, aut,or o!

9DS0-optimi#er, :9N /:Ta%les**li%iptc, /:T;-9nal3#er

:atc,es accepted into

0inu1 kernel, iproute&, ipta%les, li%pcap and .ires,ark

4rgani#er o! Net!ilter .orks,op &'(<

3/36
DDoS protection using Netfilter/iptables
.,at will 3ou learn= .,at will 3ou learn=

0inu1 Kernel is vulnera%le to simple S>N attacks

End-,ost mitigation?s alread3 implemented in kernel

s,ow it is not enoug,

Kernel* serious @listen@ socket scala%ilit3 pro%lem

solution is stalled """ ,ow to work-around t,is

$irewall-%ased solution* s3npro13 Aipta%les-net!ilterB

How !ast is state!ul !irewalling

.,ere is our pain points

0earn Net!ilter tricks* %oost per!ormance a !actor ('

4/36
DDoS protection using Netfilter/iptables
$irst* Basic N/ tuning ('( $irst* Basic N/ tuning ('(

9ll tests in presentation

Basic tuning

$irst kill CirD%alanceE

N/ ,ardware Dueue, are :2 aligned

Disa%le Et,ernet !low-control

/ntel i1g%e ,w-driver issue

single %locked ,w Dueue %locks ot,ers

$i1 in kernel v<"F"' commit <e%e7!de%' Ai1g%e* Set DropGEN %it

w,en multiple R1 Dueues are present w-o !low controlB
5/36
DDoS protection using Netfilter/iptables
$ocus* $looding DoS attack $ocus* $looding DoS attack

Denial o! Service ADoSB attacks

$ocus* T: !looding attacks

9ttacking t,e <-Wa3 HandS,ake A<.HSB

End-,ost resource attack

S>N !lood

S>N-9K !loods

9K !loods A<
rd
packet in <.HSB

9ttacker o!ten spoo!s src /:

Descri%ed in R$ )57H*
T: S>N $looding 9ttacks and ommon Iitigations
6/36
DDoS protection using Netfilter/iptables
0inu1 current end-,ost mitigations 0inu1 current end-,ost mitigations

Jargon R$ )57H AT: S>N $looding 9ttacks and ommon IitigationsB

0inu1 uses ,3%rid solution

S>N Ccac,eE

Iini reDuest socket

Iinimi#e state, dela3 !ull state alloc

S>N C%acklogE o! outstanding reDuest sockets

9%ove limit, use S>N CcookiesE

7/36
DDoS protection using Netfilter/iptables
Details* S>N @cac,e@ savings Details* S>N @cac,e@ savings

Small initial TB ATransmission ontrol BlockB

struct reDuestGsock Asi#e F6 %3tesB

mini sock to represent a connection reDuest

But alloc si#e is ((& %3tes

S09B %e,ind ,ave si#eo!Astruct tcpGreDuestGsockB

Structs em%edded in eac,-ot,er

F6 %3tes JJ struct reDuestGsock

7' %3tes JJ struct inetGreDuestGsock

((& %3tes JJ struct tcpGreDuestGsock

$ull TB Astruct inetGsockB is 7<& %3tes

Anote, si#es will increase-c,ange in more recent kernelsB
8/36
DDoS protection using Netfilter/iptables
Details* /ncreasing S>N %acklog Details* /ncreasing S>N %acklog

Not recommended to increase !or DoS

4nl3 increase, i! legitimate tra!!ic cause log*

CT:* :ossi%le S>N !looding """E

/ncreasing S>N %acklog is not o%vious

9d8ust all t,ese*

-proc-s3s-net-ipv)-tcpGma1Gs3nG%acklog

-proc-s3s-net-core-soma1conn

S3scall listenAint sock!d, int backlogBK

9/36
DDoS protection using Netfilter/iptables
S>N cookies S>N cookies

Simpli!ied description

S>N packet

don?t create an3 local state

S>N-9K packet

Encode state in SELM Aand T: optionsB

9K packet

ontains SELMN( Aand T: timestampB

Recover state

SH9 ,as, is computed wit, local secret

;alidate A<.HSB 9K packet state

10/36
DDoS protection using Netfilter/iptables
Details* S>N-cookies Details* S>N-cookies

S>N cookies SH9 calculation is e1pensive

SNI: counters ASince kernel v<"(B

TCPReqQFullDoCookies * num%er o! times a

S>N44K/E was replied to client

TCPReqQFullDrop * num%er o! times a S>N reDuest

was dropped %ecause s3ncookies were not
ena%led"

9lwa3s on option

-proc-s3s-net-ipv)-tcpGs3ncookies J &
11/36
DDoS protection using Netfilter/iptables
So, w,at is t,e pro%lem= So, w,at is t,e pro%lem=

Oood End-Host counter-measurements

:ro%lem* 0/STEN state scala%ilit3 pro%lem

;ulnera%le !or all !loods

S>N, S>N-9K and 9K !loods

Num%ers* Peon :2 PFFF' ('O i1g%e

N4 0/STEN socket*

&"5')"(&7 pkts-sec -- S>N attack

0/STEN socket*

&F&"'<& pkts-sec -- S>N attack

<<6"FH6 pkts-sec -- S>NN9K attack

<<("'H& pkts-sec -- 9K attack

12/36
DDoS protection using Netfilter/iptables
:ro%lem* S>N-cookie vs 0/STEN lock :ro%lem* S>N-cookie vs 0/STEN lock

Iain pro%lem*

S>N cookies live under 0/STEN lock

/ proposed S>N %rownies !i1 AIa3 &'(&B

,ttp*--t,read"gmane"org-gmane"linu1"network-&<&&<7

Oot re8ected, %ecause not general solution

e"g" don?t ,andle S>N-9K and <.HS

N$.S&'(< got clearance as a !irst step solution

Need to C!orward-portE patc,es

ABug ('FH<6) - R$E* :arallel S>N cookies ,andlingB

13/36
DDoS protection using Netfilter/iptables
$irewall and :ro13 solutions $irewall and :ro13 solutions

Netork!"ase# ountermeasures

.esle3 I" Edd3, descri%es S>N-pro13

/n isco* T,e /nternet :rotocol Journal - ;olume 5,

Num%er ), &''6, link* ,ttp*--goo"gl-9(99Q

Net!ilter* ipta%les target S$NPR%&$

9vail in kernel <"(< and RHE0H

B3 :atrick IcHard3, Iartin Top,olm and Ie

9lso works on local,ost

Oeneral solution

Solves S>N and 9K !loods

/ndirect trick also solves S>NN9K

14/36
DDoS protection using Netfilter/iptables
S>N pro13 concept S>N pro13 concept
15/36
DDoS protection using Netfilter/iptables

S>N:R4P> needs conntrack

.ill t,at %e a per!ormance issue=

Base per!ormance*

&"56)"'5( pkts-sec -- N4 0/STEN sock N no ipta%les rules

&))"(&5 pkts-sec -- 0/STEN sock N no ipta%les rules

0oading conntrack* AS>N !lood, causing new conntrackB

)<F"F&' pkts-sec -- N4 0/STEN sock ' conntrack

(H&"55& pkts-sec -- 0/STEN sock ' conntrack

0ooks %ad"""

%ut / ,ave some tricks !or 3ou K-B

onntrack per!ormanceA(B onntrack per!ormanceA(B
16/36
DDoS protection using Netfilter/iptables
onntrack per!ormanceA&B onntrack per!ormanceA&B

onntrack Alock-lessB lookups are really fast

:ro%lem is insert and delete conntracks

2se to protect against S>NN9K and 9K attacks

De!ault net!ilter is in T: ClooseE mode

9llow 9K pkts to create new connection

Disa%le via cmd*

sysctl -w net/netfilter/nf_conntrack_tcp_loose=0

Take advantage o! state C/N;90/DE

Drop invalid pkts before reac,ing 0/STEN socket

iptables -m state --state INVALID -j DR!
17/36
DDoS protection using Netfilter/iptables
onntrack per!A<B 9K-attacks onntrack per!A<B 9K-attacks

(C) attacks, conntrack per!ormance

De!ault ClooseJ(E and pass /N;90/D pkts

(H5"'&H pkts-sec

0ooseJ' and and pass /N;90/D pkts

&<F"5') pkts-sec Alisten lock scalingB

0ooseJ' and and DR4: /N;90/D pkts

F"F<<"'F6 pkts-sec
18/36
DDoS protection using Netfilter/iptables
onntrack per!A)B S>N-9K attack onntrack per!A)B S>N-9K attack

S$N!(C) attacks, conntrack per!ormance

S>N-9Ks don?t auto create connections

T,us, c,anging ClooseE setting is not important

De!ault pass /N;90/D pkts Aand ClooseJ(EB

&<'"<)7 pkts-sec

De!ault DR4: /N;90/D pkts Aand ClooseJ(EB

F"<7&"&6F pkts-sec

De!ault DR4: /N;90/D pkts Aand ClooseJ'EB

F")'7"<'H pkts-sec
19/36
DDoS protection using Netfilter/iptables
S3npro13 per!ormance S3npro13 per!ormance

%nl* conntrack S$N attack proble+ left

Due to conntrack insert lock scaling

Base per!ormance*

&))"(&5 pkts-sec -- 0/STEN sock N no ipta%les rules

0oading conntrack* AS>N !lood, causing new conntrackB

(H&"55& pkts-sec -- 0/STEN sock ' conntrack

Using S$NPR%&$

,-./0-.,1 pkts-sec -- 0/STEN sock N s*npro2* N conntrack

20/36
DDoS protection using Netfilter/iptables
ipta%les* s3npro13 setupA(B ipta%les* s3npro13 setupA(B
2sing S>N:R4P> target is complicated

S>N:R4P> works on untracked conntracks

/n CrawE ta%le, CnotrackE S>N packets*
iptables -t raw -I PREROUTING -i $DEV -p tcp -m tcp --syn \
--dport $PORT -j CT --notrack
21/36
DDoS protection using Netfilter/iptables
ipta%les* s3npro13 setupA&B ipta%les* s3npro13 setupA&B

Iore strict conntrack ,andling

Need to get unknown 9Ks A!rom <.HSB to %e

marked as /N;90/D state

Aelse a conntrack is 8ust createdB

Done %3 s3sctl setting*
sbins!sctl -" netnet#iltern#$conntrack$tcp$loose%&
22/36
DDoS protection using Netfilter/iptables
ipta%les* s3npro13 setupA<B ipta%les* s3npro13 setupA<B

atc,ing state*

2NTR9KED JJ S>N packets

/N;90/D JJ 9K !rom <.HS

2sing S>N:R4P> target*
iptables -' INPUT -i $DEV -p tcp -m tcp --dport $PORT (
-m state --state INV')ID*UNTR'C+ED (
-j SYNPROXY --sack-perm --timestamp --"scale , --mss -./&
23/36
DDoS protection using Netfilter/iptables
ipta%les* s3npro13 setupA)B ipta%les* s3npro13 setupA)B

Trick to catc, S>N-9K !loods

Drop rest o! state /N;90/D, contains S>N-9K

iptables -' INPUT -i $DEV -p tcp -m tcp --dport $PORT (
-m state --state INV')ID -j DROP

Ena%le T: timestamping

Because S>N cookies uses T: options !ield

sbins!sctl -" netip0.tcp$timestamps%-
24/36
DDoS protection using Netfilter/iptables
ipta%les* s3npro13 setupAFB ipta%les* s3npro13 setupAFB

onntrack entries tuning

Ia1 possi%le entries & Iill

&77 %3tes R & Iill J FH6"' IB

netnet#iltern#$conntrack$ma1%2&&&&&&

/I:4RT9NT* 9lso ad8ust ,as, %ucket si#e

-proc-s3s-net-net!ilter-n!GconntrackG%uckets writea%le

via -s3s-module-n!Gconntrack-parameters-,as,si#e

Has, 7 %3tes R &Iill J (6 IB

ec3o 2&&&&&& 4 s!smod5len#$conntrackparameters3as3si6e
25/36
DDoS protection using Netfilter/iptables
:er!ormance S>N:R4P> :er!ormance S>N:R4P>

Script ipta%lesGs3npro13"s, avail ,ere*

,ttps*--git,u%"com-netoptimi#er-network-testing-%lo%-master-ipta%les-ip
ta%lesGs3npro13"s,

2sing S>N:R4P> under attack t3pes*

&"765"7&) pkts-sec S S>N-!lood

)"5)7")7' pkts-sec S 9K-!lood

F"6F<"(&' pkts-sec S S>NN9K-!lood

26/36
DDoS protection using Netfilter/iptables
S>N:R4P> parameters S>N:R4P> parameters

T,e parameters given to S>N:R4P> target

Iust matc, t,e %ackend-server T: options

Ianual setup A,elper tool n!s3npro13B

4nl3 one setting per rule

Not use!ul !or DH: %ased network

Future plan

9uto detect server T: options

Simpl3 allow !irst S>N t,roug,

atc, S>N-9K and decode options

ARHBQ ('F56H5 - R$E* S3npro13* auto detect T: optionsB

27/36
DDoS protection using Netfilter/iptables
Real-li!eA(B* Handle 5'' Kpps Real-li!eA(B* Handle 5'' Kpps
28/36
DDoS protection using Netfilter/iptables
Real-li!eA&B* SH9 sum e1pensive Real-li!eA&B* SH9 sum e1pensive

S>N cookie SH9 sum is e1pensive

Bug ('FH<F& - R$E* /mprove S>N cookies calculations

29/36
DDoS protection using Netfilter/iptables
Real-li!eA<B* 4ut tra!!ic normal Real-li!eA<B* 4ut tra!!ic normal
30/36
DDoS protection using Netfilter/iptables
/ssue* $ull connection scala%ilit3 /ssue* $ull connection scala%ilit3

Still e1ists* Scala%ilit3 issue wit, !ull conn

Iade it signi!icantl3 more e1pensive !or attackers

At,e3 need real ,ostsB

$uture work* !i1 scala%ilit3 !or

entral lock* 0/STEN socket lock

entral lock* Net!ilter new conntracks A.ork-in-progressB

31/36
DDoS protection using Netfilter/iptables
$i1ing central conntrack lock $i1ing central conntrack lock

onntrack issue

/nsert - delete conntracks takes central lock

.orking on removing t,is central lock

ABased on patc, !rom Eric Duma#etB

ARHBQ (')<'(& - @net!ilter* conntrack* remove t,e central spinlock@B

:reliminar3 results, S>N-!lood

No 0/STEN socket to leave out t,at issue

)<F"F&' pkts-sec S conntrack wit, central lock

("6&6"H76 pkts-sec S conntrack wit, parallel lock

32/36
DDoS protection using Netfilter/iptables
Hack* Iulti listen sockets Hack* Iulti listen sockets

Hack to work-around 0/STEN socket lock

Simpl3 0/STEN on several ports

2se ipta%les to rewrite-DN9T to t,ese ports

33/36
DDoS protection using Netfilter/iptables
Hack* $ull conn ,as,limit trickA(B Hack* $ull conn ,as,limit trickA(B

:ro%lem* $ull connections still ,ave scala%ilit3

:artition /nternet in -&) su%nets

A(&7R&F6R&F6 - &'5H(F& J ) ma1 ,as, listB

0imit S>N packets e"g" &'' S>N pps per src su%net

Iem usage* !airl3 ,ig,

$i1ed* ,ta%le-si#e &'5H(F& R 7 %3tes J (6"H IB

;aria%le* entr3 si#e (') %3tes R F''''' J F& IB

34/36
DDoS protection using Netfilter/iptables
Hack* $ull conn ,as,limit trickA&B Hack* $ull conn ,as,limit trickA&B

2sing ,as,limit as work-around

9ttacker needs man3 real ,osts, to reac, !ull conn

scala%ilit3 limit
iptables -t ra" -' PREROUTING -i $DEV (
-p tcp -m tcp --dport 7& --s!n (
-m 3as3limit (
--3as3limit-abo0e 2&&sec --3as3limit-b5rst -&&& (
--3as3limit-mode srcip --3as3limit-name s!n (
--3as3limit-3table-si6e 2&8,-92 (
--3as3limit-srcmask 2. -j DROP
35/36
DDoS protection using Netfilter/iptables
9lternative usage o! @socket@ module 9lternative usage o! @socket@ module

9void using conntrack

2se 1tGsocket module

$or local socket matc,ing

an !ilter out <.HS-9Ks Aand ot,er com%osB

:arameter --nowildcard

:ro%lem can still %e invalid-!lood 9Ks

Iitigate %3 limiting e"g",as,limit

Didn?t scale as well as e1pected

,ttps*--git,u%"com-netoptimi#er-network-testing-%lo%-master-ipta%les-ipta%lesGloc
alGsocketG,ack"s,
36/36
DDoS protection using Netfilter/iptables
T,e End T,e End

T,anks to Iartin Top,olm and 4ne"com

$or providing real-li!e attack data

Download slides ,ere*

,ttp*--people"net!ilter"org-,awk-presentations-devcon!&'()-

$eed%ack-rating o! talk on*

,ttp*--devcon!"c#-!-<H

/! unlikel3Atime !or DuestionsB

Luestions=
37/36
DDoS protection using Netfilter/iptables
E1tra Slides E1tra Slides
38/36
DDoS protection using Netfilter/iptables
Disa%le ,elper auto loading Disa%le ,elper auto loading

De!ault is to auto load conntrack ,elpers

/t is a securit3 riskT

:oking ,oles in 3our !irewallT

Disa%le via cmd*

ec"o 0 # /proc/sys/net/netfilter/nf_conntrack_"elper

ontrolled con!ig e1ample*

iptables -t raw -p tcp -p $%$% -j &' --"elper ftp

Read guide ,ere*

,ttps*--,ome"regit"org-net!ilter-en-secure-use-o!-,elpers-