492033 - Random disconnects when TCP SACK is enabled

Status: RESOLVED FIXED

(mozilla.org Graveyard :: Server Operations, task, minor)

Description

[Chris AtLee [:catlee]]

If I'm connected to the MPT VPN, and I ssh to one of the machines in the build network, e.g. moz2-linux-slave02.build.mozilla.org, I'll occasionally get randomly disconnected from the machine.

Something is actually sending me TCP RST packets:
13:51:45.781316 IP moz2-linux-slave02.build.mozilla.org.ssh > 10.2.21.38.51851: Flags [R.], seq 2701, ack 2657, win 79, options [nop,nop,TS val 6684024 ecr 4245624108,nop,nop,sack 1 {2653630087:2653630183}], length 0
13:51:46.601533 IP moz2-linux-slave02.build.mozilla.org.ssh > 10.2.21.38.51851: Flags [R.], seq 2701:2749, ack 2657, win 79, options [nop,nop,TS val 6684241 ecr 4245624108], length 48

Disabling SACK on my machine (echo 0 > /proc/sys/net/ipv4/tcp_sack) seems to fix the issue, but I'd rather not have to do that.

Also, ssh'ing to mpt-vpn.mozilla.com, and then ssh'ing to the desired machine seems to fix the issue.

Comment 1

[Derek Moore [:dmoore]]

This bug is due to the lack of TCP SACK support our Cisco firewall software.

When we first encountered it, the status would have been WONTFIX. Cisco had not released a workaround at that time. They have recently released a software upgrade, however, which disables TCP SACK negotiation during the TCP handshake. The end result is the same as disabling SACK locally.

IT will have a discussion later today to determine a schedule for upgrading our firewalls.

Comment 2

[Derek Moore [:dmoore]]

Tentatively scheduled for the evening of 05/12

Comment 3

[Derek Moore [:dmoore]]

Running on the new software version now.
Secondary firewall upgrade scheduled for 05/14.

Comment 4

[Derek Moore [:dmoore]]

Firewall upgrade with SACK workaround is complete

Comment 5

[Chris AtLee [:catlee]]

I've been hitting this again lately.  I haven't tried tcpdump to determine if I'm getting RST packets, but disabling SACK locally does seem to fix the problem.

Comment 6

[Chris AtLee [:catlee]]

I'm still hitting this when SACK is enabled locally:

08:12:02.519529 IP production-master.build.mozilla.org.ssh > 10.2.21.86.53291: Flags [R.], seq 725:749, ack 825, win 57, options [nop,nop,TS val 4631086 ecr 2744937268], length 24

Comment 7

[matthew zeier [:mrz]]

Fill me in here - we did a firewall upgrade to fix this but you're saying it's not fixed?

Comment 8

[Chris AtLee [:catlee]]

Yes, I'm still getting random disconnects from various machines.  I just managed to capture this when trying to ssh to production-master:


15:05:48.768830 IP production-master.build.mozilla.org.ssh > 10.2.21.38.60501: Flags [R.], seq 1837:1901, ack 1793, win 79, options [nop,nop,TS val 55677417 ecr 548682849], length 64

Comment 9

[Derek Moore [:dmoore]]

We've confirmed that the patch for this problem actually regressed in a subsequent firewall firmware upgrade. We're investigating our current options.

Comment 10

[Derek Moore [:dmoore]]

A subsequent upgrade has been made available to us from Cisco. We'll schedule an appropriate downtime window shortly.

Comment 11

[Derek Moore [:dmoore]]

FWSM upgrade was completed tonight, and (once again) we've enabled the TCP SACK workaround.