A redundant load-balancing firewall system, using FreeBSD.

Goal: Configure two or more redundant IPF-based firewalls, which will also act as load-balancers (henceforth referred to as "FWLBs") for an internet services cluster. If one firewall fails, the second will take over as the firewall/load balancer. Barring any catastrophes, this should result in a highly-available, almost zero downtime environment. This hopefully saves you loads of money on expensive dedicated hardware devices. We'll end up with a configuration like the following:
~~~ ( net ) ~~~ | | | ---------| switch | ---------/ \ / VIP \ ---------------| fwlb1 | | fwlb2 | ---------------\ VIP / \ / --------| switch | --------/ \ / \ ----------------| server1 | | server2 | -----------------

Where the first VIP (Virtual IP) is the publicly known IP of your site, and the second is an RFC 1918 private address that your servers will use as a default route and NAT through.

Prerequisites: - Two dual-homed FreeBSD-STABLE boxes - Two switches/hubs

runit) is not strictly necessary. FreeVRRPd is what will be handling our failover for us. if they'll be needing to initiate outbound connections. so I've used it in the examples here.conf" ipnat_rules="/etc/ipnat. add the following to /etc/rc.kernel compiled with IPFILTER and IPFILTER_LOG is a simple but flexible load balancer.nwo. but it's a tool I find quite useful for process control and supervision./usr/ports/sysutils/daemontools or /usr/ports/sysutils/runit . please refer to: http://www./usr/ports/net/pen . For more IPF info. or PF.Two or more real servers for a cluster .net/ipf/ipf-howto.conf" ipnat_flags="-CF" ipmon_enable="YES" And put your rulesets in the proper places. for that matter) ruleset.edu. The two points important to keep in mind are to make sure you allow free multicast communication between both firewalls(VRRP requires it).html Short story . IPF/IPNAT configuration is beyond the scope of this document./usr/ports/net/freevrrpd .. mkdir -p /etc/supervise/pen/log . allowing traffic in on the port of the service you'll be balancing. more licensing-friendly and featureful workalike. and to make sure other hosts can't(or you'll run into some unpleasant security possibilities). but there's been plenty written on the subject. PART 2: The balancer 1) Create the necessary users and directories.html http://coombs.build a kernel with the options above. Optionally.anu.au/~avalon/ip-filter.conf: ipfilter_enable="YES" ipfilter_rules="/etc/ipf. Pen Procedure: PART 1: The Firewall Create an IPF(or IPFW. allow the cluster to be able to NAT through this box. Daemontools(or Gerrit Pape's excellent.

This is configured for round-robin balancing .local csh -cf '/usr/local/bin/svscanboot &' sleep 5 && svstat pen . cd /service ln -s /etc/supervise/pen echo "csh -cf '/usr/local/bin/svscanboot &'" >> /etc/rc.mkdir -p /var/chroot/pen mkdir -p /var/log/pen pw useradd pen -s /bin/false -d /var/chroot/pen pw useradd penlog -s /bin/false -d /var/chroot/pen chown penlog:pen /var/log/pen 2) Create the runfiles for pen. This example has pen logging somewhat verbosely. remove the "-r" flag. It will be balancing port 80 incoming to port 80 on hostname1 and hostname2. to aid in debugging.if you require sticky sessions. 3) Start up the load-balancing services. You may wish to remove the "-d" in a production environment. cd /etc/supervise/pen cat << _EOF_ > run #!/bin/sh exec 2>&1 exec pen -d -u pen -j /var/chroot/pen -C localhost:8888 -f -r 80 hostname1 hostname2 _EOF_ chmod 755 run cd log cat << _EOF_ > run #!/bin/sh exec /usr/local/bin/setuidgid penlog /usr/local/bin/multilog s999999 n20 /var/log/pen _EOF_ chmod 755 run This will configure pen to run chrooted in /var/chroot/pen. with a control port of 8888.

touch /var/log/freevrrpd.one for the front-facing . Now.123.sample to /usr/local/etc/freevrrpd.log cat << _EOF_ >> /etc/syslog.1/32 password = vrid1 vridsdep = 2 # backend VRID [VRID] serverid = 2 interface = fxp1 priority = 255 addr = 10.111.0.0. On this machine.conf !freevrrpd *. both machines have been equal. You can confirm this by tailing /var/log/pen/current on both machines.conf.1/32 password = vrid2 vridsdep = 1 This results in 2 VRIDs being created . Copy /usr/local/etc/freevrrpd. PART 3: Redundancy 1) First.* _EOF_ /var/log/freevrrpd. configure syslog to log VRRP info to its own file. Edit the file.conf.log 2) Configure FreeVRRPd Until this point. and configure it along the following lines: # public-facing VRID [VRID] serverid = 1 interface = fxp0 priority = 255 addr = 198. you need to choose which FWLB is going to be your primary.You should now be able to point your browser. and see it balancing out to your real servers. etc to either of the IPs of these machines. dns resolver.

and verify that the connection occurred. You should now copy this file over to the slave FWLB.From one of the machines in your cluster.Watch the logs on FWLB2. This avoids the backend servers trying to route through a machine with a dead front-end connection. Note that both VRIDs depend on the other.Pull the front-end interface of FWLB1. ssh to 10. specified by the "vridsdep" field.sh start PART 4: Failover testing Now you just need to verify that this whole setup works in the case of a failure. . and log in.sh{.0. 3) Start FreeVRRPd You can now start up freevrrpd on both boxes: cp /usr/local/etc/rc.1. If another box outside of this cluster is in a position to communicate with it over VRRP. . connect to 198.111. First.While watching /var/log/pen/current on FWLB1. both VRIDs are configured to consider this host the master server during VRRP elections. . but certainly don't rely on VRRP passwords as a security measure. the other one will automatically go into backup mode.sample. port 80 from a machine on the front-end network.1. .1.} /usr/local/etc/rc.1. you've got a problem. so we have something to connect to to verify that the interfaces fail properly.SSH again to 10. port 80.d/freevrrpd.it means that if one of the interfaces in a FWLB fails.d/freevrrpd.network. configure both FWLB boxes to start an SSH daemon. and change both priority fields to be 100.0. . Verify that you see the connection occur.123. Change the password field on both to something more original. You should see the hostname of FWLB2 in the SSH banner. This is important . and one for the rear-facing one that the cluster will be using.111.123. Verify the hostname of the machine is the hostname of the master FWLB. In this example.0. Try the following scenarios: .0. Connect to 198. failing both interfaces to the slave FWLB.

just do: penctl localhost:8888 server $servername blacklist 1 This will reset the blacklist timeout to 1 second. Simply add the hostname to the end of the pen command. bringing it back into the pool.html http://siag. For fun.html http://smarden.to/daemontools.. Permanently adding or removing servers to/from a pool: In the case that more servers need to be added or removed. and do: svc -t /service/pen This will TERM and restart pen. you can just do: penctl localhost:8888 server $servername blacklist 99999 This will blacklist the server for 99999 seconds.bsdshell.html http://www.org/runit . Notes: Removing servers from a pool: Pen cannot permanently remove servers from a pool. Now conduct the same tests when unplugging the backend interface.yp. While this may not cause interruption to the user. the /service/pen/run file will need to be edited. When the server is back in service. giving adequate time to seed the server. you may want to just hit the reset button on FWLB1 while actively hitting the web servers.org/rfcs/rfc2338. References: http://www. but if you need to have it ignore a server while doing upgrades or such.faqs. it's probably wise to do this while in maintenance mode or off hours.Reconnect the front-end interface of FWLB1.net/hut_fvrrpd. Verify that both interfaces on FWLB1 recover back to master state.nu/pen/ http://cr.

redundancy.© 2004 David Thiel --.lx [@ at @] redundancy.org Updated Sun Mar 21 18:23:28 PST 2004 .

343!41 4:7890 ..07       070901789'! '79:.3#57.43/8.//70889.:974:90.8.3/9080.078:80./01.! 8905:-. 420/700$ $%-408  %489.3/%974:    !70706:8908   %4/:.90.07807.9 4:7807.08.    807.

:-8 .

:8907  .807. %447247070.078147..

:87.

54798.

309.

503  .

:87.

54798.

309.

1700.775/  .

:87.

54798.

88:98.

0243944847./.

:87.

54798.

88:98.

904:9-4:3/  .:890794-0.7 -:99 8.   70.7:39  0730.94413/6:90:801:147574..0 7:39 8349897..07147:8 .4250/9!%#. .0039 2470.3! 47! 47! 1479.50 80..43974.490.07 700'##!/8.92.3/34:7 1.4.0383 1703/.04: -0-.90.9907 7:0809 .02439448 470779!.0:80/93900.3/10.3.9438 !.4330.3439054794190807./-.930.-094%974:  98-4 190 -0300/39439.2508070   !74.0/:70  !#%%070.3/!%#*     !038.3   5943.9:701: 47.088.43  97.9-0.3/8:507.843 84 .8250-:910-04.3.11.088 .

422:3..3 9 474: 7:33948420:350.3994005323/.8.943-09003  -49170.8 '##!706:7089 .:2039 -:99070 8-0035039799034390  8:-0.3/942..89.417002:9.9438-043/908.3980.08:7049074898  .9 %0945439825479.807010794  995.08:704:.!%.7094  2.450  4198/4..431:7.:795488-908   472470!314 50.

.

 34 309.

51.

51 494 92 995.

.

442-8 .3: 0/: ..:.

43.=...

0730990459438.0 .-4.//901443 94.5 1907 92  $4798947 -:/.

09.

7.431  51907*03.-0$ 51907*7:08. .

09.

431 53.9*7:08.51 .

09.

088.-0$  3/5:94:77:080983905745075.7:8078..909030.07   70.9*1.431 53..3//70.3.08   !#%%0-.9 .53.8  5243*03.94708   2/7 5.

09.

8:507.80.

503.

4 .

2/7 5.

.7..

.7449.

503 2/7 5.

..7.

4.

503 5:807.//503 8.

-3.

80 /.1.

.7..

.7449.

503 5:807.//5034 8.

-3.

80 /.1.

..7.

7449..

435034503.503 .

..7.

4.

/.90907:3108147503   .503   70.

09.

80.8:507.

503 ..9* *7:3 .

-3.

  00.8  00.503 / :503 .

.7..

.7449.

9* *7:3 .20  * * ./4  .24/7:3 .503 4..489 1 7 4893..20 4893.

-3.

.8  00.

:87.

.4..

-3.

809://5034.

:87.

4...

-3.

2:9483  .

..7.

4.

74490/3.503  * * .24/7:3   %8.431:70503947:3.

..7.

.7449.

94303.3.20.79:5904.3 14:706:7089. 8088438 7024.090 /3.43974547941 9-0-. %80.3807.503  9.8503 438420.3.3/4893.7432039    $9.423945479 434893..090 71.574/:./ -. 8947024./.9.08   ..35479  3.20 %88 ..07-480 94.431:70/14774:3/ 74-3-.3./3/0-:3 4:2..250..

.0 3 8.807.

09.

8:507.80.

503 0.4.1 .8 .

:87.

4...

-3.

3-449 ..8.8.

09.

4..8 .7. .1 .

:87.

.4..

-3.

89.3-449  8005 8.9503  ..8.8.

 9409074190!84190802.4317298-9.807.3...07 09.3.3 .3/8009-..078 4:.3 4:9944:770.308 . 4:84:/34-0.-09454394:7-74807 /387084.

7...

4.

503.

431:70884944'##!31494984310   94:...:7703943-492..308   !#%#0/:3/.3.   789 .

..7.

4.

.1700.775/ 4 .9* *.

09.

431  1700.775/  .884 .

..7.

4.

308.30 45..4480.1700..0-00306:.775/ 4  * *    431:70700'##!/  &39985439 -492..7  3 982. 4 4: 300/94.84394-04:7572.

:87.

.4..

09.

431 8.25094 .775/ .1700.

:87.

.4..

09.

. 1.775/ ..1700.3'# '#( 807.015  57479 .43901443308  5:-.07/ 39071.//7   .431 /99010 .431:70 9.3/.

03/'# '#( 807..7/ .7/8/05  -. 5.//7 ..8847/.015 57479 .07/ 39071.

 5.90/ 4301479017439 1.8847/.3 .7/ .70..7/8/05  %8708:983'#8-03.

8.54894394.30905./0.1.33489..34309.3/43 9  7043'##!5.431:70/94.91430 419039071. -:9.4394-.990.8847/8.79700'##!/  4:.5..775/43-49-408  .079.39 920.3/ .30947 .389.250 -49'#8.049..10/-90 .309.078973 9474:90974:..574-02    $9.943   4:84:/34.:8907 -0:83 3980.08 94908.34907-4  4:98/04198.:890783.422:3.2.8847/  10/43-49948420932470473.9..8 904907430 ..07/:73'##!00.:7920.4330.79:51700..70.0794908.083..8:70 1.3/4301479070.7/8/0510/ %8825479.9438   4909..909  94..0 .3-4939071.438/07 98489902.:942.7 1.0 %8..80.07'##! 4: ..4598104.03/807..4/890-.:524/0 1./17439 03/.9-49'#8/0503/43904907 850.30-495747910/894-0 .8907807.

:87.

4...

09.

/.7.

250 < .1700.775/ 8 8.

:87.

.4..

09.

7. /.

1700..0719.994  94.03.4.30890 4893..084209394.8041.2041902.79.3/43 '071904893.431:70-49-408  9489.8907   0.0243 840.99840809:54783  90.3.081.775/ 889.:8907 8894   .3$$/.30834:7.9.2041902..4330.748   74243041902.574507 %790  14438.79   !#%.0719..:70 789 .1.0790893   44::89300/94..99039071.

.7..

4.

503.

041   .3/.990.904843 4330.9434.:7    !:9017439 03/39071.994    5479  .4330.3307  ..4330.9.2041 390$$-..:770/   $$.4330.0719.394  4:84:/800904893.2.994     5479 1742..30439017439 03/ 30947 '0719..94:80090.9434...:7703943 .

.90   4. 39490544   !072.070/43:57.0807.544  !03.078    4908  #024.4330./08478:.07 0390807.3039.0993900-807.899204:99480.43/8 .//3477024.078-..544 -:914:300/ 94.9908.4....03/39071. 4:.3994:8999070809-:994343 0.041 '0719..89  %8-.0 :89/4  503.90920 94800/90807.9 -4939071.0781742..99017439 03/39071.07807.942.07-.890789.0714780.  #0..89  %87080990-....073.94.30397024.093470..073.3 :89/4  503./06:.3807..3807..94..209089803:35:390-.43/ -7339-.20-.07807.3807.33495072.084370.807..9..07894.20-.43/:.0  471:3 4:2..489807.0781742..3.489807.8990807.

.078300/94-0.0/ 90 .809.//0/477024.92470807.544  390.1742.

0..807.

503.

.422.20949003/4190503.//90 4893. 9.7:310300/94-00/90/ $25.3/ .3//4  8.

.0.807.

3/7089..349.3903.-8094/498032.79503 0982.:8039077:5943 9490:807 9 8574-.0 24/0474114:78    #010703.503  %8%#.3.08   995.

.

68 47. 1.

8.71.

 92  995.71.

.

 -8/80 309.

:9*1.775/ 92  995.

.

8. 3:.

503.

  995.

.

.7 5 94.

02439448 92  995./.

.

7/03 47.82.

7:39  .

3.3..90/$:3. 47  &5/./%0 .7!$%   . 70/:3/.9(70/:3/.  .