758852 - crash @ nsHttpConnectionMgr::nsHalfOpenSocket functions in Private Browsing

Status: VERIFIED FIXED

(Core :: Networking: HTTP, defect)

Description

[Scoobidiver (away)]

Several crash signatures implying nsHttpConnectionMgr::nsHalfOpenSocket functions appeared in 15.0a1/20120525070245. The regression range might be (because of several nightlies per day, debug symbols are sometimes missing):
http://hg.mozilla.org/mozilla-central/pushloghtml?fromchange=f43e8d300f21&tochange=3871d6ca5fb2

Here are some comments:
"Crashes only in Private Browsing Mode. For now I can reproduce it in safe mode, but not in a new profile"
"Again, crash in Private Browsing Mode. This time I had 1 tab at youtube.com. It crashed when I try to open the addon manager."
"It is not the userscript's problem, but definitely related to Private Browsing Mode."

It's likely a regression from bug 722845.

Here are the first frames of various stacks:
Frame 	Module 	Signature 	Source
0 		@0xe1013141 	
1 	xul.dll 	nsHttpConnection::Init 	netwerk/protocol/http/nsHttpConnection.cpp:143
2 	xul.dll 	nsHttpConnectionMgr::nsHalfOpenSocket::OnOutputStreamReady 	netwerk/protocol/http/nsHttpConnectionMgr.cpp:2566
3 	xul.dll 	nsSocketOutputStream::OnSocketReady 	netwerk/base/src/nsSocketTransport2.cpp:490
4 	xul.dll 	nsSocketTransport::OnSocketReady 	netwerk/base/src/nsSocketTransport2.cpp:1532
5 	xul.dll 	nsSocketTransportService::DoPollIteration 	netwerk/base/src/nsSocketTransportService2.cpp:741
6 	xul.dll 	nsSocketTransportService::Run 	netwerk/base/src/nsSocketTransportService2.cpp:612
7 	xul.dll 	nsThread::ProcessNextEvent 	xpcom/threads/nsThread.cpp:624
8 	xul.dll 	nsThread::ThreadFunc 	xpcom/threads/nsThread.cpp:257

Frame 	Module 	Signature 	Source
0 	xul.dll 	nsHttpConnectionMgr::RestrictConnections 	netwerk/protocol/http/nsHttpConnectionMgr.cpp:1191
1 	xul.dll 	nsHttpConnectionMgr::nsHalfOpenSocket::~nsHalfOpenSocket 	netwerk/protocol/http/nsHttpConnectionMgr.cpp:2349
2 	xul.dll 	nsHttpConnectionMgr::nsHalfOpenSocket::Release 	netwerk/protocol/http/nsHttpConnectionMgr.cpp:2320
3 	xul.dll 	nsRefPtr<nsIRunnable>::~nsRefPtr<nsIRunnable> 	obj-firefox/dist/include/nsAutoPtr.h:874
4 	xul.dll 	nsSocketTransport::OnSocketDetached 	netwerk/base/src/nsSocketTransport2.cpp:1663
5 	xul.dll 	nsSocketTransportService::DetachSocket 	netwerk/base/src/nsSocketTransportService2.cpp:181
6 	xul.dll 	nsSocketTransportService::DoPollIteration 	netwerk/base/src/nsSocketTransportService2.cpp:752
7 	xul.dll 	nsSocketTransportService::Run 	netwerk/base/src/nsSocketTransportService2.cpp:612
8 	xul.dll 	nsThread::ProcessNextEvent 	xpcom/threads/nsThread.cpp:624
9 	xul.dll 	nsThread::ThreadFunc 	xpcom/threads/nsThread.cpp:257

Frame 	Module 	Signature 	Source
0 	xul.dll 	nsHttpConnectionMgr::nsHalfOpenSocket::OnTransportStatus 	netwerk/protocol/http/nsHttpConnectionMgr.cpp:2711
1 	xul.dll 	nsSocketTransport::SendStatus 	netwerk/base/src/nsSocketTransport2.cpp:882
2 	xul.dll 	nsSocketTransport::OnSocketConnected 	netwerk/base/src/nsSocketTransport2.cpp:1382
3 	xul.dll 	nsSocketTransport::OnSocketReady 	netwerk/base/src/nsSocketTransport2.cpp:1550
4 	xul.dll 	nsSocketTransportService::DoPollIteration 	netwerk/base/src/nsSocketTransportService2.cpp:741
5 	xul.dll 	nsSocketTransportService::Run 	netwerk/base/src/nsSocketTransportService2.cpp:612
6 	xul.dll 	nsThread::ProcessNextEvent 	xpcom/threads/nsThread.cpp:624
7 	xul.dll 	nsThread::ThreadFunc 	xpcom/threads/nsThread.cpp:257

Frame 	Module 	Signature 	Source
0 	xul.dll 	nsHttpConnectionMgr::nsHalfOpenSocket::SetupStreams 	netwerk/protocol/http/nsHttpConnectionMgr.cpp:2375
1 	xul.dll 	mozilla::CalibratedPerformanceCounter 	xpcom/ds/TimeStamp_windows.cpp:521
2 	xul.dll 	nsCOMPtr<nsIContentSecurityPolicy>::StartAssignment 	obj-firefox/dist/include/nsCOMPtr.h:809
3 	xul.dll 	nsHttpConnectionMgr::nsHalfOpenSocket::SetupBackupStreams 	netwerk/protocol/http/nsHttpConnectionMgr.cpp:2470
4 	xul.dll 	nsHttpConnectionMgr::OnMsgProcessPendingQ 	netwerk/protocol/http/nsHttpConnectionMgr.cpp:1930
5 	xul.dll 	xul.dll@0xb83ab 	
6 	xul.dll 	nsHttpConnectionMgr::nsHalfOpenSocket::Notify 	netwerk/protocol/http/nsHttpConnectionMgr.cpp:2555
7 	xul.dll 	nsTimerImpl::Fire 	xpcom/threads/nsTimerImpl.cpp:476
8 	xul.dll 	nsTimerEvent::Run 	xpcom/threads/nsTimerImpl.cpp:556
9 	xul.dll 	nsThread::ProcessNextEvent 	xpcom/threads/nsThread.cpp:624

More reports at:
https://crash-stats.mozilla.com/report/list?signature=nsHttpConnection%3A%3AInit%28nsHttpConnectionInfo*%2C+unsigned+short%2C+nsISocketTransport*%2C+nsIAsyncInputStream*%2C+nsIAsyncOutputStream*%2C+nsIInterfaceRequestor*%2C+nsIEventTarget*%2C+unsigned+int%29
https://crash-stats.mozilla.com/report/list?signature=nsHttpConnectionMgr%3A%3ARestrictConnections%28nsHttpConnectionMgr%3A%3AnsConnectionEntry*%29
https://crash-stats.mozilla.com/report/list?signature=nsHttpConnectionMgr%3A%3AnsHalfOpenSocket%3A%3AOnTransportStatus%28nsITransport*%2C+unsigned+int%2C+unsigned+__int64%2C+unsigned+__int64%29
https://crash-stats.mozilla.com/report/list?signature=nsHttpConnectionMgr%3A%3AnsHalfOpenSocket%3A%3ASetupStreams%28nsISocketTransport**%2C+nsIAsyncInputStream**%2C+nsIAsyncOutputStream**%2C+bool%29

Comment 1

[Fanolian]

After numerous crashes, I find that it is easier to reproduce the crash by setting browser.cache.disk.enable and network.http.use-cache to false, restart the browser, then enter private browsing mode.
While (re)connecting youtube.com frontpage (I press ENTER in the url bar instead of using the reload button), open addon manager using the firefox button. Sometimes Nightly crashes before the addon manager shows. I happened to crash Nightly thrice in a new profile by these steps. However it is very crashy in my main profile.

My latest crash report from a new profile:
https://crash-stats.mozilla.com/report/index/bp-bbff74df-6d49-49e9-a2b9-e63d32120526

Comment 2

[Fanolian]

There is a much simpler way to reproduce the crash.

1. In a new profile, set browser.cache.disk.enable and network.http.use-cache to false, restart the browser.
2. Enter private browsing mode, simply visit http://www.nba.com/playoffs/2012/index.html.

If the page finishes loading, reload it again by the reload button. Nightly will crash eventually after a few tries.

Comment 3

[Honza Bambas (:mayhemer)]

We forget to set the private flag on mConnectionInfo in nsHttpChannel before we request speculative connect.

CI the null transaction uses is then modified on the main thread in nsHttpChannel::SetupTransaction while speculative connect is in nsHttpConnectionMgr::GetOrCreateConnectionEntry on the socket thread.  It looks up the entry in mCT, but doesn't find it.  So it creates a new one and puts it to mCT.  But, the HashKey() of the CI has changed right between those two operations and (unexpectedly) an existing entry is replaced (i.e. the old one is released) with a new one.  Then nsHalfOpenSocket holds ref to a broken mEnt.


This also exposes a more wide bug about nsConnectionInfo object not implemented as as thread-safe but used as thread-safe.

Comment 4

[Honza Bambas (:mayhemer)]

Attachment: v1

Comment 5

[Patrick McManus [:mcmanus]]

Comment on attachment 627768 [details] [diff] [review]
v1

Review of attachment 627768 [details] [diff] [review]:
-----------------------------------------------------------------

confirmed that every setanonymous() now has a setprivate()

Comment 6

[Honza Bambas (:mayhemer)]

Comment on attachment 627768 [details] [diff] [review]
v1

https://hg.mozilla.org/integration/mozilla-inbound/rev/737025a86de9

Comment 7

[Ed Morley [:emorley]]

https://hg.mozilla.org/mozilla-central/rev/737025a86de9

Comment 8

[Paul Silaghi, QA [:pauly]]

No crash loading the STR in comment 2.
Verified fixed on FF 15b3 on Win 7/64, Ubuntu 12.04 and Mac OS X 10.6.