258054 - iptables eventually causes severe packet loss on iguana

Status: RESOLVED FIXED

(mozilla.org Graveyard :: Server Operations, task)

Description

[Myk Melez [:myk] [@mykmelez]]

When iptables runs on iguana to firewall that server, it eventually starts
dropping most legitimate packets to the machine, making the machine virtually
nonresponsive.  Restarting iptables fixes the problem temporarily.  It seems to
take about a day for the problem to reoccur.

Comment 1

[Myk Melez [:myk] [@mykmelez]]

I've restarted iptables, so it should be fast again, and I created a cronjob
that restarts iptables on the hour, which should keep the problem at bay until
we can definitively solve it (probably by setting up a separate dedicated
firewall machine).

Comment 2

[Myk Melez [:myk] [@mykmelez]]

cc:ing others who might be able to help.

The problem is that with iptables enabled on a busy RHEL AS 3.0 server, most
legitimate packets get dropped after about a day.  I'm not sure if it builds
over time or if packets just suddenly start getting dropped.  In both reported
cases, update.mozilla.org, which iguana hosts, was very slow a day after
starting iptables, and all connections to iguana, including SSH and ping
connections, saw massive packet loss (80% in the first case, 96% in the second).
 Restarting iptables immediately fixed the problem.

I now have a cron job running that restarts iptables every hour.  This should
work around the problem until we can find its cause and fix it.

Comment 3

[Scott Kveton]

Justdave spent some quality time last night working on this.  It appears that
the default firewall script that ships with RHEL 3 has some problems that are
causing these issues.  Justdave ported over the iptables script from mecha and
hacked it up enough to work on iguana.  It works and traffic has been steady but
it needs to be tweaked to no close established connections.

Comment 4

[Myk Melez [:myk] [@mykmelez]]

When I woke up this morning, iguana was responding to about 90% of my pings. 
After restarting iptables, the machine started responding to 100% of my pings. 
So there may still be a problem.

Comment 5

[Dave Miller [:justdave]]

We have not seen any recurrance of these issues in the last two months that I
know of, and iptables is still running.  If anyone has still been seeing
problems, please reopen this.