Firewall hijinx

Posted in: Tech

I've been neck-deep in iptables trying to set up a dual redundant firewall out of a pair of Linux 2.6 boxes. I'm currently moving from a six year long fuzzy phase of vaguely understanding how iptables works to rapidly upgrading that into a solid working knowledge: enough to actually create something non-trivial, on my own. Put another way, enter an endless loop of typing commands, reading documents, (note the order there), puzzling at why it doesn't work...

So in the course of all this something quite technically funny just happened...

Still at the stage of quite early set-up. To get the basic address translation and routing sorted, I'm testing with one of the fw boxes pretending to be a server in the DMZ. Thus external requests are translated and forwarded from the fw to the server, and server requests are similarly translated back out.

So in order to test this I set up a forward (DNAT) from fw:80 to server:80 and tried to hit it from outside. Not working. After a fair amount of time in the aforementioned methodology of semi-mindful experimentation... success! I must've tried about every possible thing including a variety of IP changes, switches, deletions including to find out which of the nine interfaces I was dealing with was disconnected (two of them apparently).

Now, in order to reduce complexity I had at some stage dropped the ports aspect of the incoming translation - i.e. make the rule wider to translate everything from fw to server. In other words, the fw disappears to new connections! Now, I had my existing ssh connection kept alive; had I lost that I'd've been SOL with no access to the fw. I hadn't realised all this yet....

Back to the story: Having figured out exactly which of the various things I did had impacted exactly which aspects of the various connections I was working on I started to put the rules into the FWBuilder config I'd built. All sorted, I uploaded it to the fw. Except the SSH connection used to upload the firewall config was forwarded to the server! D'oh! Server promptly vanishes to the network having been firewalled off with wrong IPs.

Shit. I'll have to call the NOC and get a powercycle (and run risk of fsck barfing) or wait 'til I'm next down there. Shit.

But no! By some ridiculously improbable fluke a another machine in the cabinet had the fw's IP stored in its arp cache with the server's MAC address - bing, bypass forwarding and we're in!

Firewall:

porus:~# ifconfig | grep -A 1 eth0
eth0 Link encap:Ethernet HWaddr 00:E0:81:60:C3:EB
inet addr:217.207.14.103 Bcast:217.207.14.255 Mask:255.255.255.0

Random local cabinet host:

stix:/tmp# arp 217.207.14.103
Address HWtype HWaddress Flags Mask Iface
217.207.14.103 ether 00:E0:81:60:E1:E7 C eth2

Same IP, different MAC

Lesson learnt: I would've caught this forward from fw to the server by having a different password or key on each. Bad me, the two machines are virtually identical, down to having same root passwords.

Universal lesson repeated for the millionth time: give up after hours at it and get some sleep ;-)

Posted by Paul Makepeace at April 20, 2005 01:32 | TrackBack