529698 - New crash [@nsBaseChannel::OnStartRequest(nsIRequest*, nsISupports*)] due to msntoolbar

Status: RESOLVED FIXED

(Core :: Networking, defect)

Description

[Johnny Stenback (:jst)]

A new crash in nsBaseChannel::OnStartRequest() started appearing on November the 17th. The crash is heavily correlated with the silverlight plugin, and even more so with the extensions:

    100% (93/93) vs.   1% (1003/103169) [e-mail] (4.0)
    100% (93/93) vs.   1% (1053/103169) {27182e60-b5f3-411c-b545-b44205977502} (1.0)

This has showed up both in 3.5.5 and on trunk. The stacks appear to have this in common:

0xblah (a constant blah, per version, that is)
nsBaseChannel::OnStartRequest  	 netwerk/base/src/nsBaseChannel.cpp:663
nsInputStreamPump::OnStateStart 	netwerk/base/src/nsInputStreamPump.cpp:439
nsInputStreamPump::OnInputStreamReady 	netwerk/base/src/nsInputStreamPump.cpp:395
nsInputStreamReadyEvent::Run 	xpcom/io/nsStreamUtils.cpp:191
nsThread::ProcessNextEvent 	xpcom/threads/nsThread.cpp:527
NS_ProcessNextEvent_P 	obj-firefox/xpcom/build/nsThreadUtils.cpp:250
...

Not necessarily anything we can do about this, but I wanted to get this on file...

Comment 1

[Johnny Stenback (:jst)]

Actually this was seen first on 20091105, so it's been around for a while, but it seems like it's getting more common...

Comment 2

[Jonas Sicking (:sicking) No longer reading bugmail consistently]

So here's what the stack looks like when opening a minidump:

00820218()	
nsDocumentOpenInfo::OnStartRequest(...) Line 268
nsBaseChannel::OnStartRequest(request=0x00852470, ctxt=0x00000000) Line 665
nsInputStreamPump::OnStateStart() Line 443
nsInputStreamPump::OnInputStreamReady(stream=0x01809c08) Line 395
nsInputStreamReadyEvent::Run() Line 112
nsThread::ProcessNextEvent(mayWait=1, result=0x0012fca8) Line 522
nsBaseAppShell::Run()
nsAppStartup::Run()

Note that the argument values aren't really reliably at all. I removed the arguments to OnStartRequest because they were clearly bogus.

The nsDocumentOpenInfo frame is calling |request->GetStatus(&status);|, and it looks like it's somewhere in that call that we're crashing. So a good guess is that there's a misbehaving protocol handler involved here and that the request object comes from that handler. If it was our handler we should have symbols for the last frame.

Comment 3

[Jonas Sicking (:sicking) No longer reading bugmail consistently]

Possibly what happened here is that hotmail put a link on their site that uses a custom protocol provided by the msntoolbar. That would explain the sudden jump in crashes.

Comment 4

[Andy Rivas]

I'm not sure I'm the right assignee here but I'll look to find the right person for this bug.

Comment 5

[Julie Nierenberg]

Can you please provide more information for this error?  A crash dump would be very helpful.
Thanks

Comment 6

[timeless]

andy: thanks :)

comment 2 indicates sicking had a minidump. At this point we're trying to save at least one minidump per signature iiuc.

Comment 7

[Johnny Stenback (:jst)]

We do have minidumps for this crash, but we are not sharing those with third parties since they could contain privacy sensitive information. It looks to me though that the minidumps we have for this particular crash doesn't really show much more than what's in comment 2 in this bug, so I'm not sure sharing a minidump would actually benefit Microsoft in this particular case.

FWIW, the crash looks to be coming from us attempting to call GetStatus() on an nsIRequest implementation which either crashes or the request pointer is bad.

Comment 8

[Jonas Sicking (:sicking) No longer reading bugmail consistently]

I suspect the actual request pointer is good. We've already called QueryInterface on it without crashing, and if it was a null pointer we would have crashed inside nsDocumentOpenInfo::OnStartRequest rather than one frame after it.

Comment 9

[Johnny Stenback (:jst)]

Yup, you're right, the pointer looks ok, it's just crashing in its GetStatus() call.

Comment 10

[Andy Rivas]

Can I resolve this bug or assign to someone else?

Comment 11

[Jonas Sicking (:sicking) No longer reading bugmail consistently]

Well, there's not really that much we can do on the firefox side here. The crash still seems to be in msntoolbar code related to protocol handlers. It started appearing with no apparent change on the firefox end.

Data is also indicating that a lot of crashes are happening on the livemail website (jst, this is correct, right?).

My *guess* is that the livemail website started using a protocol implemented by the msntoolbar more, which caused the crash to show up higher in our crash statistics.


I believe that we are looking into if we can get you guys our minidumps. However given how sensitive data they contain, it's likely this unfortunately won't be possible. It's something we can discuss though a different channel than this bug.

Comment 12

[Julie Nierenberg]

Please email me directly to discuss.  [e-mail]

Comment 13

[timeless]

andy: sorry, i should have adjusted the assignee earlier. it's hard for me to keep track of all of the ms projects which overlap with our code.

julie: does your code implement nsIRequest? and does it do something interesting in GetStatus() ?

Comment 14

[d]

Should we blocklist the previous and current versions of the MSN toolbar until a fix is released? Judging by the comments at the crash reports, people are starting to get angry.

Comment 15

[Jonas Sicking (:sicking) No longer reading bugmail consistently]

This there were close to 560 crashes on the 3.5.5 branch over the past week (750 total all branches). Doesn't put it close to the top 100. Would be interesting to know how many users we have using the msntoolbar to get a sense for how many we'd affect by blocklisting the toolbar, vs. how many we'd help with fewer crashes.

To be clear here, we won't blocklist the toolbar without talking with microsoft first.

Comment 16

[d]

I see, and yes, of course Microsoft should be consulted first. Personally I've seen people with the toolbar, but never seen anyone of those who actually use it. They belong to the group of people who doesn't know why it is there, what is does, or how to remove/hide it.

Comment 17

[Julie Nierenberg]

I believe the crashes associated with this bug all occur in Win NT.  See the link below.
The MSN Toolbar does not get installed on Win NT (XP, Vista, Win 7 only).  THerefore, I don't believe this issue is caused by MSN Toolbar.

http://crash-stats.mozilla.com/report/list?product=Firefox&version=Firefox:3.5.5&platform=windows&query_search=signature&query_type=contains&query=nsBaseChannel&date=&range_value=1&range_unit=weeks&do_query=1&signature=nsBaseChannel::OnStartRequest(nsIRequest*, nsISupports*)&page=2

Comment 18

[Jonas Sicking (:sicking) No longer reading bugmail consistently]

"Windows NT 5.x" is the same as Windows XP and "Windows NT 6.x" is the same as Windows Vista, no? I these are simply identifiers reported to us through some windows API.

I'm fairly sure firefox 3.5 doesn't even run on Windows NT 4.

Comment 19

[Julie Nierenberg]

FYI - The MSN Toolbar team is actively attempting to reproduce the crash and to fix it.

Comment 20

[Julie Nierenberg]

FYI - We can reproduce a Firefox browser crash on the hotmail site, but the crash call stack does not have msntoolbar code.  We have NOT been able to reproduce the crash in which the msntoolbar code is in the call stack.   We are still working on a repro in which the toolbar code in the call stack.

See the following callstack from a crash dump where the 3.5.6 FF crashes on the hotmail site:

001bb398 53b35b6b 001bb45c 00000000 0fe7f800 xul!GCGraphBuilder::NoteXPCOMChild+0x8f [e:\builds\moz2_slave\win32_build\build\xpcom\base\nscyclecollector.cpp @ 1447]
001bb3c4 53bc2748 5444acbc 0fe7f800 001bb45c xul!nsDocument::cycleCollection::Traverse+0x2bb [e:\builds\moz2_slave\win32_build\build\content\base\src\nsdocument.cpp @ 1800]
001bb3e0 53b5464e 5444acbc 0fe7f800 001bb45c xul!nsHTMLDocument::cycleCollection::Traverse+0x15 [e:\builds\moz2_slave\win32_build\build\content\html\document\src\nshtmldocument.cpp @ 240]
001bb444 53bce01c 01d0a800 00000000 01dcf000 xul!nsCycleCollector::MarkRoots+0xde [e:\builds\moz2_slave\win32_build\build\xpcom\base\nscyclecollector.cpp @ 1571]
001bb4a8 53bd95e2 01d0a800 00000000 01d27a00 xul!nsCycleCollector::BeginCollection+0x57 [e:\builds\moz2_slave\win32_build\build\xpcom\base\nscyclecollector.cpp @ 2515]
001bb4b8 5f25b0ce 01d27a00 00000002 01d27a00 xul!XPCCycleCollectGCCallback+0x3b [e:\builds\moz2_slave\win32_build\build\js\src\xpconnect\src\nsxpconnect.cpp @ 390]
001bb560 5f2a529c 01d27a00 00000000 00000000 js3250!js_GC+0x28e [e:\builds\moz2_slave\win32_build\build\js\src\jsgc.cpp @ 3504]
001bb574 53bf1f6c 01d27a00 01d27a00 53bd95a7 js3250!JS_GC+0x4c [e:\builds\moz2_slave\win32_build\build\js\src\jsapi.cpp @ 2458]
001bb630 53be99d0 53c18e32 00000001 00000002 xul!nsXPConnect::Collect+0x75 [e:\builds\moz2_slave\win32_build\build\js\src\xpconnect\src\nsxpconnect.cpp @ 478]
001bf4e0 53c43fca 01d0a800 00000001 53c43ff8 xul!nsCycleCollector::Collect+0x94 [e:\builds\moz2_slave\win32_build\build\xpcom\base\nscyclecollector.cpp @ 2386]
001bf4ec 53c43ff8 53da4e6a 53da4f35 00000001 xul!nsCycleCollector_collect+0x11 [e:\builds\moz2_slave\win32_build\build\xpcom\base\nscyclecollector.cpp @ 3046]
001bf4f0 53da4e6a 53da4f35 00000001 53c18e74 xul!nsJSContext::CC+0x2a [e:\builds\moz2_slave\win32_build\build\dom\src\base\nsjsenvironment.cpp @ 3512]
001bf524 53b389ee 134180b0 01d1d5b0 001bf800 xul!nsJSContext::MaybeCC+0x19b345
001bf548 53b1f71a 00000001 00000001 001bf568 xul!nsThread::ProcessNextEvent+0x1be [e:\builds\moz2_slave\win32_build\build\xpcom\threads\nsthread.cpp @ 522]
001bf560 53c3535b 00000001 1000bde0 53ac5f5f xul!nsBaseAppShell::Run+0x4a [e:\builds\moz2_slave\win32_build\build\widget\src\xpwidgets\nsbaseappshell.cpp @ 169]
001bf56c 53ac5f5f 01dc3340 01d20098 00000000 xul!nsAppStartup::Run+0x1e [e:\builds\moz2_slave\win32_build\build\toolkit\components\startup\src\nsappstartup.cpp @ 194]
001bf5d8 6c53fccd 00000000 5431d358 5431d8a0 xul!XRE_main+0xe49 [e:\builds\moz2_slave\win32_build\build\toolkit\xre\nsapprunner.cpp @ 3323]
01d204c0 00000000 02240000 00000001 02250000 mozcrt19!_close+0x11d [e:\builds\moz2_slave\win32_build\build\obj-firefox\memory\jemalloc\src\close.c @ 65]

Comment 21

[Julie Nierenberg]

The crash has been fixed and a new version of the Bing Bar is available at http://www.discoverbing.com/toolbar/ 

Over the next several weeks we will be updating users.

Comment 22

[Jonas Sicking (:sicking) No longer reading bugmail consistently]

Thanks Julie!

Marking this one FIXED then! Hopefully we'll be able to see the number of crashes dropping in our crash stats.