Tuesday, September 16, 2008

Building a redundant bridging firewall

Having a firewall is a good thing to have. Having a bridging firewall is even better. Unlike with a regular firewall, bridging firewall requires no change to the internal hosts/clients. It can be connected and disconnected any time without any notice from the clients. A firewall can be a single point of failure bringing your network down. Having redundancy on the firewall part is an important factor.

Building a redundant firewall requires 2 computers. Each computer will have 2 NICs. We will be using RedHat Linux but any flavor of linux will work. Here's a diagram of where the firewall will fit in your network:

Redundant means that if your primary firewall fails your secondary firewall will continue the flow of the traffic. Firewall 1 will be the primary and will be doing most of the network duty if it fails(no longer bridges traffic) then Firewall 2 will take over.

There are several issues to address when having multiple firewalls. Each machine will be keeping seperate logs and if you need to investigate your would need to check both firewalls. Another issue is to monitor these firewalls, that way if your primary firewall fails and secondary takes over you should be immediately notified. We use Big Brother monitoring, but any monitoring program suite will do. Interesting fact to know, is the network flow. All network packets will go from the Internet to Firewall 1 and then inside. If you are accessing say Firewall 2 and even though Firewall 2 has a direct Internet connection packets would still go through Firewall 1, inside, and then to Firewall 2...this is because of the way the Spanning Tree Protocol(multiple bridges) work. If you want to read up on the linux bridging go to this link: LinuxBridging.pdf

Linux 2.4 kernel comes with bridging installed but in order to control bridging(use iptables) a patch would need to be applied. You can get the patch from http://bridge.sourceforge.net. You would also need to install bridge utilities.

Make sure you turn off your networking rc scripts from starting your network devices, etc: for redhat make sure you remove /etc/sysconfig/network-scripts/ifcfg-eth0 and eth1. In /etc/rc.d/rc.local put /etc/rc.d/rc.bridge. We will use a seperate script to start our network devices. This is how our /etc/rc.d/rc.bridge will look like:

brctl addbr br0
brctl setbridgeprio br0 0 # Sets this bridge as master bridge
brctl sethello br0 1
brctl setmaxage br0 2
brctl addif br0 eth1
brctl addif br0 eth0
ifconfig eth1
ifconfig eth0
route add default gw YOURGATEWAYADDRESS br0
Your bridge will now be working, now do the same thing with your firewall 2. I've setup an rsync script to mirror both firewalls that way changes on one firewall would propogate to other just by running "firewall.sync".

We will use iptables. One can write the rules in a flat file and then execute them, etc: /etc/sysconfig/iptables. Having a graphical iptables rules configurator would be better especially if there's a lot of hosts behind the firewall. A graphical config utility allows you to visualise and objectify your firewall. We use fwbuilder, fwbuilder can be downloaded from fwbuilder.org. Fwbuilder builds an iptables shell script that you could place in /etc/rc.d/rc.local.

The following documentation will be more specific to our site setup. The save file will be saved in /root/fwbuilder as firewall.xml. Fwbuilder will build Firewall.fw in that directory. This file will be symlinked from /etc/rc.d/Firewall.fw to /root/fwbuilder/Firewall.fw. /etc/rc.d/rc.bridge is the bridge setup. rc.firewall has starts the fwbuilder rules also adds custom iptables rules for accounting. The accounting rules are there for web page statistics. It also contains rules for restarting the mysql server. Mysql server is there for Snort IDS. The final line starts Snort.

Snort is a software IDS that monitors the network for any malicious packets if they are found it will log it. Snort can be found on snort.org Snort is installed from source in /usr/local/snort. Snort config files are in /usr/local/snort/etc. Main config file is snort.conf and rules are in ./rules directory. Snort is supplemented with a guardian perl program. This program is ran after snort starts. It monitors snort log files and anytime it detects an alert it will block the offender. It will ignore internal alerts for policy reasons. A mention on /etc/rc.d/readfw, its a perl program that reads the iptables rules and writes to /etc/guardian.ignore , the file then lets guardian know to ignore any alerts coming from inside the firewall.

There is also a perl program that monitors snort rules. Every Monday, /usr/local/snort.update/update.pl -m runs which checks whitehats.com for new rules. If any rules are found it emails the sysadmins. The sysadmin will then login to the firewall and run /usr/local/snort.update/update.pl -u to update snort rules with the new ones. If sysadmin disapproves some of the updates they will put the rule ID # in /usr/local/snort.update/arachnids.ignore.

There are some firewall policies that we have in place. Servers that users have no login access will be blocked completely from outside and will only be allowed to be accessed by the sysadmin workstations. All sysadmin logins will be logged. Usually most people log all drop packets, for us this creates a huge log file which is not only a burden to check constantly but also worthless because noone will bother to check it everyday. We will only log specific things.

The firewalls contain 2 disks. Each disk contains 3 partitions. One partition is for swap, next partition is for / and third partition is for logs. There's a weekly dd of the second partition. The third partition is not as important and thus we will not dd and instead combine the third partition from 2nd disk into larger lvm partition. This partition will be mounted on /logs.

Firewalls are synced together by a /usr/local/bin/firewall.update perl program. Config files for the program can be found on /usr/local/etc. It uses rsync to sync the hosts. Do not run this program every time there's change. After you change something make sure that it works for awhile and then run firewall.sync.