1622859 - RCWN causes half-open connections leading to IP blocks for SYN flooding

Status: NEW

(Core :: Networking: Cache, defect, P3)

Description

[Damian Yerrick]

Attachment: Wireshark capture of getting blocked in two page views

The Race Cache With Network feature caused me to see 30-minute outages on one website I often visit because its behavior looked like a SYN flood to its firewall.

Environment:

Firefox 73.0.1 or 74.0 on Xubuntu 18.04 LTS on Dell Inspiron 11-3168 laptop with Pentium N3710 CPU and Samsung SATA SSD

Steps to reproduce (for non-members):

1. Open Wireshark and tell it to capture host forums.nesdev.com
2. Open Firefox and in about:config, set network.http.rcwn.enabled to its default value of true. This must be done through about:config because of Bug 1618200.
3. Open Firefox's network monitor (Ctrl+Shift+E)
4. Browse around on forums.nesdev.com, starting at a page listing all 23 smilies available for use in new posts, which are images smaller than 500 bytes https://forums.nesdev.com/viewtopic.php?f=15&t=19841
5. Keep going for several minutes, waiting no more than 5 seconds between clicks. For example, open several pages in tabs using middle-click or Ctrl+click. Watch for cached (raced).
6. Repeat the test with network.http.rcwn.enabled set to false.

For in-depth testing:

1. Create an account and verify its email address.
2. Open Quick links > Unread posts, and open several pages in new tabs.
3. Open Test Forum, create a new topic, and press Preview several times. https://forums.nesdev.com/viewforum.php?f=15

Observed behavior:

In some cases, the Transferred column shows 10 or more cached (raced). For each of these, Wireshark shows a TCP conversation consisting only of a SYN from your machine, a SYN-ACK from the remote machine, and a RST from your machine. Each RCWN represents an embryonic half-open connection that never becomes fully open, something that also happens in a SYN flood attack.

If more than 30 embryonic connections are made in a 10-second period, all connections to nesdev.com, forums.nesdev.com, and wiki.nesdev.com from the same IPv4 address begin to time out and continue to time out for 30 minutes until the firewall's automatic block on your IPv4 address expires. A connection that becomes fully open (SYN, SYN-ACK, ACK) is not counted in the 30.

Expected behavior:

Firefox ACKs the connections, allowing them to become fully open. Or Firefox doesn't attempt RCWN more than a handful of times per registrable domain name per minute. Or Firefox doesn't attempt RCWN if most come back cached (raced). Or Firefox doesn't attempt RCWN on a machine with a SATA or NVMe SSD (and hence quick access time) and Linux (and hence no intrusive antivirus).

Additional links:

My thought process while tracking down the culprit for this network misbehavior
https://forums.nesdev.com/viewtopic.php?f=13&t=19804

Someone else recommending turning off RCWN to limit excess SYNs
https://stackoverflow.com/q/55708231/2738262

This may be the same problem that the reporter of bug 1451951 was seeing.

Comment 1

[Kershaw Chang [:kershaw]]

I think we should only establish 6 TCP connections per host regardless of network.http.rcwn.enabled being enabled or not.
Or maybe I am wrong. Michal, what do you think?

Comment 2

[Michal Novotny [:michal]]

When we race and the cache wins we cancel the network request at https://searchfox.org/mozilla-central/rev/d6f957415cf009995ecb539ef1425316d82164a9/netwerk/protocol/http/nsHttpChannel.cpp#5297. Maybe when the request wasn't sent yet it might cause resetting half-open connection. Dragana, is this possible? If yes, maybe we should close half-open connections cleanly instead of sending RST.

Comment 3

[Dragana Damjanovic [:dragana]]

I am not sure it that will help, but we can try it.

Comment 4

[Kenny Robinson]

Attachment: same problem

Thanks from progwarhuathe1973.