iptables eventually causes severe packet loss on iguana

RESOLVED FIXED

Status

mozilla.org Graveyard
Server Operations
RESOLVED FIXED
14 years ago
3 years ago

People

(Reporter: myk, Assigned: myk)

Tracking

Details

(Assignee)

Description

14 years ago
When iptables runs on iguana to firewall that server, it eventually starts
dropping most legitimate packets to the machine, making the machine virtually
nonresponsive.  Restarting iptables fixes the problem temporarily.  It seems to
take about a day for the problem to reoccur.
(Assignee)

Comment 1

14 years ago
I've restarted iptables, so it should be fast again, and I created a cronjob
that restarts iptables on the hour, which should keep the problem at bay until
we can definitively solve it (probably by setting up a separate dedicated
firewall machine).
(Assignee)

Updated

14 years ago
Blocks: 258049
(Assignee)

Comment 2

14 years ago
cc:ing others who might be able to help.

The problem is that with iptables enabled on a busy RHEL AS 3.0 server, most
legitimate packets get dropped after about a day.  I'm not sure if it builds
over time or if packets just suddenly start getting dropped.  In both reported
cases, update.mozilla.org, which iguana hosts, was very slow a day after
starting iptables, and all connections to iguana, including SSH and ping
connections, saw massive packet loss (80% in the first case, 96% in the second).
 Restarting iptables immediately fixed the problem.

I now have a cron job running that restarts iptables every hour.  This should
work around the problem until we can find its cause and fix it.

Comment 3

14 years ago
Justdave spent some quality time last night working on this.  It appears that
the default firewall script that ships with RHEL 3 has some problems that are
causing these issues.  Justdave ported over the iptables script from mecha and
hacked it up enough to work on iguana.  It works and traffic has been steady but
it needs to be tweaked to no close established connections.
(Assignee)

Comment 4

14 years ago
When I woke up this morning, iguana was responding to about 90% of my pings. 
After restarting iptables, the machine started responding to 100% of my pings. 
So there may still be a problem.
We have not seen any recurrance of these issues in the last two months that I
know of, and iptables is still running.  If anyone has still been seeing
problems, please reopen this.
Status: NEW → RESOLVED
Last Resolved: 14 years ago
Resolution: --- → FIXED
Product: mozilla.org → mozilla.org Graveyard
You need to log in before you can comment on or make changes to this bug.