When iptables runs on iguana to firewall that server, it eventually starts dropping most legitimate packets to the machine, making the machine virtually nonresponsive. Restarting iptables fixes the problem temporarily. It seems to take about a day for the problem to reoccur.
I've restarted iptables, so it should be fast again, and I created a cronjob that restarts iptables on the hour, which should keep the problem at bay until we can definitively solve it (probably by setting up a separate dedicated firewall machine).
cc:ing others who might be able to help. The problem is that with iptables enabled on a busy RHEL AS 3.0 server, most legitimate packets get dropped after about a day. I'm not sure if it builds over time or if packets just suddenly start getting dropped. In both reported cases, update.mozilla.org, which iguana hosts, was very slow a day after starting iptables, and all connections to iguana, including SSH and ping connections, saw massive packet loss (80% in the first case, 96% in the second). Restarting iptables immediately fixed the problem. I now have a cron job running that restarts iptables every hour. This should work around the problem until we can find its cause and fix it.
Justdave spent some quality time last night working on this. It appears that the default firewall script that ships with RHEL 3 has some problems that are causing these issues. Justdave ported over the iptables script from mecha and hacked it up enough to work on iguana. It works and traffic has been steady but it needs to be tweaked to no close established connections.
When I woke up this morning, iguana was responding to about 90% of my pings. After restarting iptables, the machine started responding to 100% of my pings. So there may still be a problem.
We have not seen any recurrance of these issues in the last two months that I know of, and iptables is still running. If anyone has still been seeing problems, please reopen this.
Status: NEW → RESOLVED
Last Resolved: 14 years ago
Resolution: --- → FIXED
Product: mozilla.org → mozilla.org Graveyard
You need to log in before you can comment on or make changes to this bug.