System: fail2ban and iptables

Around the beginning of 2005 we saw an increase in brute-force ssh attacks - people or robots trying different combinations of username and password to log into remote servers. A quick search on this topic returns many references to iptables and ipchains but noone really explained how they work. Having just gone through this learning curve myself, and found a satisfactory solution in the fail2ban package, I'm going to try and explain how to achieve the simple goal of banning IP addresses that make repeated failed ssh login attempts. If you want more technical information regarding firewalls and iptables in particular, see the References section at the bottom of this page.
Note: The following relates to the default config file provided with fail2ban 0.6.0 in the Debian unstable distribution and should be used for information purposes only. We take no responsibility for any consequences of following the instructions on this page.

1. What is/are iptables?
iptables is the firewall administration program for the Netfilter firewall mechanism which is built into the Linux kernel. To view your current 'firewall chains' you need to run the following command as root:
# iptables -L

This will show a list of 'chains' called INPUT, FORWARD and OUTPUT - these are always present - followed by any custom chains. The default (blank) settings will look something like:
Chain INPUT (policy ACCEPT) target prot opt source Chain FORWARD (policy ACCEPT) target prot opt source Chain OUTPUT (policy ACCEPT) target prot opt source

destination

destination

destination

If that's what you're seeing then there are no existing firewall rules and it's probably safe to proceed. Otherwise you may want to check with the sysadmin who installed any existing firewall rules before making changes.
Note: you can also use iptables -L INPUT to list just the INPUT chain. Other useful options are to add -n to display values in numerical format or -v for verbose mode. Finally iptables -h will display all command-line options.

2. Installing and configuring fail2ban
The fail2ban package is available under Debian/unstable and also as a download for other Linux systems. See the Fail2Ban website linked under Resources at the bottom of the page for details. To install on Debian:
# apt-get -t unstable install fail2ban

If you run this command then fail2ban will be installed and already running as a daemon. However you might want to edit the configuration file and stop/start the daemon to get it running how you want. The configuration file can be found at /etc/fail2ban. conf. The configuration file is broken up into sections. Most entries don't need to be changed but there are a few that you might want to edit. The DEFAULT settings apply to all sections:
[DEFAULT] maxfailures = 3 bantime = 900 findtime = 600

This says that after three failed access attempts are detected from a single IP address within 600 seconds or 10 minutes (findtime), then that address will be automatically blocked for 900 seconds (bantime). In most cases you won't see them again as the bots will move on to another server.
[MAIL] enabled = true to = root@localhost

Needs to be set to true for emails to be sent - to the root user in this case. Only do this if you're actually going to check the root mailbox otherwise you're just wasting disc space.
[Apache] enabled = false

For now we're not going to worry about monitoring the Apache log files.
[SSH] enabled = true logfile = /var/log/auth.log

fail2ban will monitor the auth.log file for failed access attempts. As soon as the daemon is running your ssh port (22) will be protected from brute-force attacks - preventing more than a small number of attempts at one time. To check if it's running:
# /etc/init.d/fail2ban status Status of fail2ban: fail2ban is running. Note: these commands will differ for different flavours of Linux.

To stop/start the daemon after making configuration changes:
# /etc/init.d/fail2ban stop Stopping fail2ban: .done # /etc/init.d/fail2ban start Starting fail2ban: .done

Actions taken by the daemon are logged by default in /var/log/fail2ban.log and you can change the verbosity in the conf file to one of: 0 - WARN, 1 - INFO or 2 - DEBUG.

3. What does fail2ban do with iptables?
This code runs when the daemon is started and adds new firewall rules using iptables:
fwstart = iptables -N fail2ban-ssh iptables -A fail2ban-ssh -j RETURN iptables -I INPUT -p tcp --dport ssh -j fail2ban-ssh

Translation:
q q q

create a NEW chain called fail2ban-ssh; APPEND a RETURN command to the end of the fail2ban-ssh chain (same as return in PHP or other languages); INSERT a rule at the start of the built-in INPUT chain that redirects SSH packets to the fail2ban-ssh chain.

This adds the following to the output of iptables -L:
Chain INPUT (policy ACCEPT) target prot opt source fail2ban-ssh tcp -- anywhere Chain fail2ban-ssh (1 references) target prot opt source RETURN all -- anywhere

destination anywhere

tcp dpt:ssh

destination anywhere

Because there are no actual rules in the fail2ban-ssh chain, connection attempts using SSH are simply redirected from the INPUT chain to the fail2ban-ssh chain and then sent straight back. What it does mean however is that we (or the fail2ban daemon to be precise) can insert new rules at any time and they will be applied to all incoming SSH traffic. The fwend commands simply reverse this process, removing the JUMP command and and then FLUSHing and DELETEing the fail2banssh chain:
fwend = iptables -D INPUT -p tcp --dport ssh -j fail2ban-ssh iptables -F fail2ban-ssh iptables -X fail2ban-ssh Note: You should always stop fail2ban before editing the config file - so that it cleans up existing rules.

To ban an IP address the following command is run, with <ip> replaced by the actual IP address or hostname captured by the failregex regular expression (see below):
fwban = iptables -I fail2ban-ssh 1 -s <ip> -j DROP

Translation:
q

Insert a rule as line 1 of the fail2ban-ssh chain to DROP all packets from the listed IP address or hostname.

When a fwban event is triggered, the output of iptables -L fail2ban-ssh --line-numbers becomes:
Chain fail2ban-ssh (1 references) num target prot opt source 1 DROP all -- <host> 2 RETURN all -- anywhere

destination anywhere anywhere

where <host> is the problem ip address or domain. While this rule is in effect all ssh requests from <host> will be silently dropped. If a second fwban even occurs while the first is in place it will appear as line 1 and the other commands moved down the chain. The fwunban command simply removes the DROP command from the chain. It's called after 10 minutes or whatever time you have set for the bantime in fail2ban.conf:
fwunban = iptables -D fail2ban-ssh -s <ip> -j DROP

4. The failregex regular expression
The most complicated part of the configuration file is the regular expression that is applied to auth.log to determine what to count as an access failure.
failregex = : (?:(?:Authentication failure|Failed [-/\w+]+) for(?: illegal user)?|Illegal user|Did not receive identification) .* from (?P<host>\S*) Note: the regular expression used here has been optimised for the auth.log in Debian Linux (Sarge) and may need to be modified slightly for other platforms.

In short, this matches any/all of the following auth.log entries and triggers a fwban event with the ip address or domain (host) that's been captured by the regular expression:
Authentication failure for username from <host> Failed password for illegal user username from <host> Failed password for username from <host> Failed keyboard-interactive/pam for username from <host> Illegal user username from <host> Did not receive identification string from <host>

For anyone trying to reverse-engineer the regular expression:
q q

(?:.*) is a non-grouping version of regular parentheses; and (?P<name>.*) is similar to regular parentheses, but the substring matched by the group is accessible via the symbolic group name name.

The single matched variable <host> will be added to the fail2ban-ssh chain if maxfailures or more matches occur within findtime. It's that simple. You will see a lot of different (and usually simpler) failregex patterns that don't contain the <host> capture. The reason for having the named capture in the regexp is to counter a bug that allows malicious users to trick your server into blocking non-hostile addresses: "fail2ban's approach to identifying an IP address in a login failure line is to scan the line for all IP addresses. Since it is possible to generate false logins from accounts such as 10.2.28.2, it is possible to force fail2ban to block access to addresses which are not attempting to connect to the system. For each IP address available to the attacker, a desired ip address may be blocked."

5. Customising the Config file
From reading more about iptables in the Linux Firewalls book (very informative but tough reading) and the online resources listed below it appears that some of the iptables commands contain options that are redundant:

q

q

the RETURN command in fwstart is not necessary as a user-defined chain will automatically return to the parent chain after processing all commands (unless another JUMP is encountered). The only advantage I see in including the RETURN command is that iptables with the -v option can then show you both the number of packets/bytes sent to and returned from fail2ban-ssh the difference telling you how many were dropped. the rule number 1 in fwban is also redundant as commands will by default be INSERTed at the top of the chain.

The fwstart command then becomes just:
fwstart = iptables -N fail2ban-ssh iptables -I INPUT -p tcp --dport ssh -j fail2ban-ssh

It's also possible to use the LOG feature in iptables rather than relying on the fail2ban program. This gives the advantage of recording the details of packets that are DROPped while a ban rule is in place. This involves a change to the fwban command:
fwban = iptables -A fail2ban-ssh -s <ip> -j LOG --log-prefix "Fail2Ban: " --log-ip-options --logtcp-sequence --log-tcp-options iptables -A fail2ban-ssh -s <ip> -j DROP Note: with the RETURN command removed we can use APPEND instead of INSERT here.

Translation:
q q

add a DROP command for <ip> at the start of the chain; add a LOG command with all options enabled at the start of the chain (before the DROP command).

and to fwunban:
fwunban = iptables -D fail2ban-ssh -s <ip> -j DROP iptables -D fail2ban-ssh -s <ip> -j LOG --log-prefix "Fail2Ban: " --log-ip-options --logtcp-sequence --log-tcp-options Note: DROP commands must exactly match the INSERT command syntax (or specify a rule number). The order isn't really important but you'll notice that we add the LOG command first and remove it last.

Now your logfiles will record all details of the dropped packages - typically in kern.log or syslog depending on your server configuration. Be aware that this can cause those files to grow quite quickly so only add the LOG commands if you're going to use the data collected.

6. Does it work?
Does it ever!?! Here you can see the results for a single server - you can tell when fail2ban was installed and why it was necessary:
$ zcat /var/log/auth.log* | grep 'Failed password' | grep sshd | awk '{print $1,$2}' | sort | uniq c 408 77 12153 9390 1033 3743 4167 2789 9200 15742 281 7 17 3 4 5 4 9 11 9 Dec Dec Dec Dec Dec Dec Dec Dec Dec Dec Dec Dec Dec Dec Jan Jan Jan Jan Jan Jan 11 12 13 14 18 19 20 25 26 27 28 29 30 31 1 2 4 5 6 7

There's word of some big changes in the fail2ban package including 'better configuration files' and 'templates for common services' among other things but it's not clear when that might make it into package form.
Note: if you just want to rate-limit SSH connections without installing new software then you can use the recent module but be aware that this can use a lot of system resources on busy networks as it has to store state information for all connections.

7. Related Articles
q q q

fail2ban and iptables fail2ban and sendmail Monitoring the fail2ban log

8. References
q q q q q

Fail2Ban iptables Tutorial Manpage of IPTABLES Linux Firewalls, Third Edition - ISBN: 0672327716 Python: Regular Expression HOWTO

System: fail2ban and sendmail
Following on from the article on fail2ban and iptables this article looks at changing the configuration file so that we can monitor the mail.log file and take action against suspect connections over smtp (port 25). Again, some of the files/methods described here may be specific to Debian (Sarge).

1. Changes to fail2ban.conf
To start with we need to create (and enable) a new section in /etc/fail2ban.conf giving instructions on where to find the logfile and what do look for:
[SMTP] enabled = true logfile = /var/log/mail/mail.log

When the daemon starts we need to go through a similar process as for blocking SSH break-in attempts. The fwstart commands create a new chain called fail2ban-mail and add a JUMP command to the INPUT chain. The fwend commands have the effect of removing both the JUMP command and the new chain: The fwcheck command simply lets fail2ban know how to check whether the fail2ban-mail chain exists.
fwstart = iptables -N fail2ban-mail iptables -A INPUT -p tcp --dport 25 -j fail2ban-mail fwend = iptables -D INPUT -p tcp --dport 25 -j fail2ban-mail iptables -F fail2ban-mail iptables -X fail2ban-mail

fwcheck = iptables -L INPUT | grep -q fail2ban-mail

Now to the 'active' components. When a match is found for the failregex expression (see below). The fwban command adds a rule to the fail2ban-mail chain instructing iptables to REJECT all packets from the relevant host. The fwunban command removes this rule. The timeregex and timepattern are the same as for auth.log:
fwban = fwunban = timeregex = iptables -A fail2ban-mail -p tcp -s <ip> -j REJECT --reject-with tcp-reset iptables -D fail2ban-mail -p tcp -s <ip> -j REJECT --reject-with tcp-reset \S{3}\s{1,2}\d{1,2} \d{2}:\d{2}:\d{2}

timepattern = %%b %%d %%H:%%M:%%S

Finally we come to the regular expression. There are a number of mail.log entries that you might want to monitor but it's tricky to cover them all with a single regexp. The regexp presented here:
failregex = [[](?P<host>\S*)[]] (?:did not issue|[(]may be forged[)])

will match lines in the mail log matching either of the following:
... [<host>] did not issue MAIL/EXPN/VRFY/ETRN during connection to MTA ... ... [<host>] (may be forged) ...

All you have to do now is stop/start the daemon as described in the previous article.

2. Changes for fail2ban 0.8
After a long wait, fail2ban now supports the use of multiple regular expressions in a single rule, making all our lives that much easier. There is also a whole new configuration system which I won't go into here as it's already quite well documented. Here is a sample failregex that you might want to use for the sendmail filter:
failregex = \[<HOST>\] .*to MTA \[<HOST>\] \(may be forged\) \[<HOST>\], reject.*\.\.\. Relaying denied

Note the use of the new <HOST> predefined entity, which matches either a hostname or an IPv4 address. These regular expressions will match and block any/all of the following:

... ... ... ...

lost input channel from [<HOST>] to MTA after data ... [<HOST>] did not issue MAIL/EXPN/VRFY/ETRN during connection to MTA ... [<HOST>] (may be forged) ... [<HOST>], reject.*... Relaying denied ...

There are also some helpful command-line tools for testing your regular expressions:
# fail2ban-regex /var/log/mail.log "\[<HOST>\], reject.*\.\.\. Relaying denied"

You can start/stop individual jails while fail2ban is still running:
# fail2ban-client reload sendmail

And you can query most aspects of the configuration:
# fail2ban-client get sendmail failregex The following regular expression are defined: |- [0]: \[(?:::f{4,6}:)?(?P<host>[\w\-.^_]+)\] .*to MTA |- [1]: \[(?:::f{4,6}:)?(?P<host>[\w\-.^_]+)\] \(may be forged\) `- [2]: \[(?:::f{4,6}:)?(?P<host>[\w\-.^_]+)\], reject.*\.\.\. Relaying denied

3. Related Articles
q q q

fail2ban and iptables fail2ban and sendmail Monitoring the fail2ban log

System: Monitoring the fail2ban log
Following on from the article on fail2ban and iptables this article looks at the fail2ban logfile and ways to analyse it using simple command-line tools such as awk and grep.

1. Format of the Logfile
At the simplest logging level, entries will appear in /var/log/fail2ban.log as follows (fail2ban version 0.8.3):
... 2006-02-13 2006-02-13 2006-02-13 2006-02-13 2006-02-13 2006-02-13

15:52:30,388 15:59:29,295 16:07:31,183 16:14:29,530 16:56:27,086 17:11:27,833

fail2ban.actions: fail2ban.actions: fail2ban.actions: fail2ban.actions: fail2ban.actions: fail2ban.actions:

WARNING WARNING WARNING WARNING WARNING WARNING

[sendmail] Ban XXX.66.82.116 [sendmail] Ban XXX.27.118.100 [sendmail] Unban XXX.66.82.116 [sendmail] Unban XXX.27.118.100 [ssh] Ban XXX.136.60.164 [ssh] Unban XXX.136.60.164

This is all very interesting, but what if you want to see a summary report so that you can try to identify IP addresses that regularly trigger Fail2Ban - so that you can send a report to their ISP or block them using a firewall script for example?

2. Generating Simple Reports
All of the following commands can be run at the command-line or via a script. They are written for Linux/UNIX systems but may work on other platforms.
Grouping by IP address: awk '($(NF-1) = /Ban/){print $NF}' /var/log/fail2ban.log | sort | uniq -c | sort Note: the variable NF equals the number of fields in each row of the logfile. So $NF is the value of the last field.

Sample output:
... 4 XXX.124.81.130 5 XXX.248.175.246 8 XXX.29.45.142

Remember that each time an IP address gets banned it's because they've been caught at least maxfailure times, so a total of 8 represents maybe 30 matches in the relevant logfile. Once they reach 10-20 you might consider them as candidates for reporting, or a more permanent solution (see below). To run this report for all logfiles only a slight change is needed:
zgrep -h "Ban " /var/log/fail2ban.log* | awk '{print $NF}' | sort | uniq -c

or, even better, we can truncate the IP addresses to identify the most problematic subnets:
zgrep -h "Ban " /var/log/fail2ban.log* | awk '{print $NF}' | awk -F\. '{print $1"."$2"."}' | sort | uniq -c | sort | tail

This is the best report for identifying problem subnets. The output will be the first two bytes of the most 'caught' subnets:
... 75 83.110. 90 219.95. 154 210.213.

Let's take the last one on the list (highlighted) and see what it's been up to:
zgrep -c 210.213. /var/log/fail2ban.log*

The output shows how many times those numbers appear in each logfile:
fail2ban.log:39 fail2ban.log.1.gz:129 fail2ban.log.2.gz:55 fail2ban.log.3.gz:78 fail2ban.log.4.gz:22

and which specific IP addresses are involved:
zcat /var/log/fail2ban.log* | awk '(NF == 6 && $NF ~ /^210\.213\./){print $NF}' | sort | uniq -c

The output of this will be a list of the IP addresses starting with 210.213. If they look like they're part of a subnet (or multiple subnets) you can copy the lowest and highest numbers in our Subnet Calculator to give you the subnet code which you can then add to your firewall rules (see below for details).
Grouping by IP address and Hostname:

The command for including hostnames in the list is a bit more complicated. You also need to insert the correct path for the logresolve program which converts IP addresses to hostnames (the path may be something like /usr/sbin/logresolve but it varies between systems):
awk '($(NF-1) = /Ban/){print $NF,"("$NF")"}' /var/log/fail2ban.log | sort | logresolve | uniq -c | sort Note: The logresolve command can take some time, especially if there are a lot of IP addresses to be processed.

The output is similar to what we've seen previously, but also includes the hostname which makes it easier to identify the ISP and/or country of origin and to see which entries might be related:
... 4 XXX.net.pk (XXX.83.169.221) 5 XXX.248.175.246 (XXX.248.175.246) 8 XXX.example.com.au (XXX.29.45.142)

You can of course just run dig, nslookup or logresolve manually on the addresses that you want to identify.
Group by IP address and Fail2Ban section: grep "Ban " /var/log/fail2ban.log | awk -F[\ \:] '{print $10,$8}' | sort | uniq -c | sort

This shows us which services each IP address has been trying to access/exploit:
... 4 XXX.124.81.130 [sendmail] 5 XXX.248.175.246 [sendmail] 8 XXX.29.45.142 [sendmail]

Now you know which logfiles to look in to see what they were doing to get banned. In this case it's most likely passing forged mail headers to sendmail which you can see in /var/log/mail/mail.log (or the relevant file on your system).
Reporting on 'today's activity:

Here's a report I find useful to run before midnight each day to generate a summary of the day's activity:
grep "Ban " /var/log/fail2ban.log | grep `date +%Y-%m-%d` | awk '{print $NF}' | sort | awk '{print $1,"("$1")"}' | logresolve | uniq -c | sort

The output will be the same as the second report above, but limited to just today's activity rather than the whole logfile.
Grouping by Date and Fail2Ban section

This report scans all fail2ban logfiles and gives you a summary of how many ban events there were for each section on each day:
zgrep -h "Ban " /var/log/fail2ban.log* | awk '{print $5,$1}' | sort | uniq -c

This can give you an idea of longer-term trends and the effectiveness of your firewall rules. This method of examining all logfiles rather than just the current one can also be applied to most of the reports above.

3. Banning an IP block or subnet
If it turns out that a significant portion of 'unwanted' traffic comes from a single ISP then you should try sending an email to their abuse address, but don't be too hopeful of getting a response. If the abuse continues then it's time to get strict. First have a look at the different IP addresses that are being caught. See if you can identify which ones come from the same subnet. The whois reports often include this information, otherwise you can use our Subnet Calculator to help you along - just paste the lowest and highest addresses into the form and it will give you the smallest subnet that covers them both.

Once you have this value (in the form XXX.XXX.XXX.XXX/XX) you can add a firewall rule using iptables to block them from the server completely, or just from the port they're abusing. For a single address you don't need to worry about subnets and the address can be used directly.
Block a subnet from accessing SSH: iptables -I INPUT -p tcp -s XXX.XXX.XXX.XXX/XX --dport ssh -j REJECT --reject-with tcp-reset Block a subnet from accessing SMTP: iptables -I INPUT -p tcp -s XXX.XXX.XXX.XXX/XX --dport smtp -j REJECT --reject-with tcp-reset

and so on for any other services. These rules will be added to the start of your firewall or you can include a rule number to decide where they appear. As always the DROP command is identical, just with -D in place of -I. To see what effect these rules are having - the number of packets and bytes being blocked by each rule - use the following command and look at the values in the first two columns.
iptables -vnL INPUT --line-numbers

At some point (hopefully) the source computer will be 'fixed' or in any case stop abusing your server. You should then remove the firewall rules.

4. Monitoring the fail2ban log with fail2ban 0.8
This is something I've been meaning to investigate for some time now, and there have been a number of request for this ability. Can we use fail2ban to block for a longer time (even permanently) addresses after they hav been blocked a number of times by the normal fail2ban filter. It seems that it is possible, though you may have to set up different jails for different ports. For example, for repeat offenders according to the sendmail filter, add the following to /etc/fail2ban.jail.local:
[fail2ban-smtp] enabled port filter logpath maxretry findtime bantime = = = = = = = true smtp fail2ban-smtp /var/log/fail2ban.log 3 21600 86400

And then create a file /etc/fail2ban/filter.d/fail2ban-smtp.conf with the following:
failregex = \[sendmail\] Ban <HOST> ignoreregex =

Finally start the new jail:
# fail2ban-client add fail2ban-smtp # fail2ban-client start fail2ban-smtp

With these settings, fail2ban will monitor it's own logfile and if a HOST is banned three times (maxretry) in six hours (findtime) they will incur a new ban lasting a full 24 hours (bantime). If the bantime value is negative then the HOST in question will never be unbanned. Similar rules can be set up for other existing jails, and they can be combined if they share the same port. Please note: this jail is still experimental so use it with care. Please let s know though the Feedback link below if you have any questions or comments about using it on your server.

5. Related Articles
q q q

fail2ban and iptables fail2ban and sendmail Monitoring the fail2ban log

Sign up to vote on this title
UsefulNot useful