878792 - orphaned CLOSE_WAIT sockets with cloudsound.com

Status: RESOLVED FIXED

(Core :: Networking: HTTP, defect)

Description

[Jonathan Jacobs]

User Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:24.0) Gecko/20130602 Firefox/24.0 (Nightly/Aurora)
Build ID: 20130602031240

Steps to reproduce:

Log into soundcloud.com (notifications are delivered to users by Websocket, this doesn't seem to happen unless one is logged in, afaict.) Observe Firefox process's TCP/IP connections (I used Process Explorer) for a few minutes.


Actual results:

The number of connections to pushers.soundcloud.com grows continuously with most connections in the CLOSE_WAIT state. Eventually SoundCloud stops behaving correctly and general browsing in Firefox is hindered to the point of things either loading extremely slowly or not at all.


Expected results:

Presumably eventually some of these CLOSE_WAIT connections should actually be closed and freed.

(FWIW, I was unable to reproduce this in Chrome 29, which means it is likely that this is a result of a Firefox issue. I'm not also not sure if this occurs on other sites that employ Websocket, this just happens to be the most apparent to me.)

Comment 1

[Jonathan Jacobs]

I should probably also mention that this was over HTTPS.

Comment 2

[Jonathan Jacobs]

I sampled a few different Nightly builds:

2013-03-31: The effect does still seem to exist, but the growth of CLOSE_WAIT connections seems fairly slow.
2013-05-12: The effect does still seem to exist and seems less pronounced than Firefox 24, but more than the Firefox 22 nightly.

(Note: My method for determining these was not terribly scientific, but involved around 15-20 minutes of regular usage, which did seem to produce different results between versions.)

Comment 3

[Jonathan Jacobs]

Sorry, Drew. I didn't actually see your changes.

Comment 4

[Patrick McManus [:mcmanus]]

I've been actively looking at this for the last day or so, thanks to a comment in 469344

The problem occurs with soundcloud's websockets server, but I really think the issue is in our generic connection handling with long timeouts.. soundcloud just happens to have a hostname with several IP addresses and most of them timing out, plus websocket code that keeps trying to connect periodically in the background.. so its an ideal trigger.

Comment 5

[Patrick McManus [:mcmanus]]

right now pushers.soundcloud.com resolves to 6 addresses.

most of them don't connect at all, occasionally one of them connects after a very long time.

Necko will only try and connect at the http (or websockets) level for 90 seconds by default.

After that timeout we do SocketTransport()->Close(NS_ERROR_NET_TIMEOUT), followed by AsyncWait(nullptr) (which means don't call me back on network events);

And that indeed forced a closed of the socket. Unfortunately the nsSocketTransport::RecoverFromError() logic will try the next DNS record silently for us on TIMEOUT.

So the socket transport service moved onto the next record, but the http connection manager was no longer listening.. if the next record actually connected (probably very slowly) there was nothing to dispatch I/O events to so the socket transport went into the idle list and never got polled again.. so when the FIN did arrive from the server (timing us out this time) we don't read it.

The fix is simple - just close with NS_ERROR_ABORT instead of NET_TIMEOUT to more accurately convey that we are aborting the socket.

This is a candidate for at least one level of backport. I'm not too worried about soundcloud (which is an ideal test case - thank you Jonathan Jacobs) but a small amount of this can accumulate across the web and theoretically lead to those unreproducible complaints of having to restart firefox because networking isn't running.

Comment 6

[Patrick McManus [:mcmanus]]

Attachment: part 2/2 - 64bit clean logging changes for socket transport

Comment 7

[Patrick McManus [:mcmanus]]

Attachment: orphaned CLOSE_WAIT sockets with multiple A records

Comment 8

[Patrick McManus [:mcmanus]]

Attachment: orphaned CLOSE_WAIT sockets with multiple A records

also update dns blacklist for all connection failures, to preserve DNS failover

Comment 9

[u408661]

Comment on attachment 770390 [details] [diff] [review]
orphaned CLOSE_WAIT sockets with multiple A records

Review of attachment 770390 [details] [diff] [review]:
-----------------------------------------------------------------

OK, looks good (as discussed via email).

Comment 10

[Patrick McManus [:mcmanus]]

remote:   https://hg.mozilla.org/integration/mozilla-inbound/rev/099ffd0d0d37
remote:   https://hg.mozilla.org/integration/mozilla-inbound/rev/d1d6724bd3a7

Comment 11

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/mozilla-central/rev/099ffd0d0d37
https://hg.mozilla.org/mozilla-central/rev/d1d6724bd3a7

Comment 12

[Evgeny]

Hi, 
I still have this problem on Mozilla Firefox 24.0 (Linux 3.9.3-gentoo #1 SMP PREEMPT Thu May 23 09:42:56 MSK 2013 x86_64).
When I have 550 connections 
$ netstat -npt |grep firefox |awk '{print $6}' |sort|uniq -c 

    549 CLOSE_WAIT
      1 ESTABLISHED

and I can't open new tab or renew an opened.

Comment 13

[Patrick McManus [:mcmanus]]

(In reply to Evgeny from comment #12)
> Hi, 
> I still have this problem on Mozilla Firefox 24.0 (Linux 3.9.3-gentoo #1 SMP
> PREEMPT Thu May 23 09:42:56 MSK 2013 x86_64).
> When I have 550 connections 
> $ netstat -npt |grep firefox |awk '{print $6}' |sort|uniq -c 
> 
>     549 CLOSE_WAIT
>       1 ESTABLISHED
> 
> and I can't open new tab or renew an opened.

the target milestone is firefox 25

Comment 14

[Evgeny]

Is this bug will fixed in an update of ESR version (FF 24) ?

Comment 15

[Patrick McManus [:mcmanus]]

(In reply to Evgeny from comment #14)
> Is this bug will fixed in an update of ESR version (FF 24) ?

no. It does not meet the criteria for backporting to ESR: "Maintenance of each ESR, through point releases, is limited to high-risk/high-impact security vulnerabilities and in rare cases may also include off-schedule releases that address live security vulnerabilities. Backports of any functional enhancements and/or stability fixes are not in scope."
 (http://www.mozilla.org/en-US/firefox/organizations/faq/)