538724 - Firefox 3.6.x + winxp topcrash starting ~Dec 30 and increasing [@ nsHttpTransaction::DeleteSelfOnConsumerThread() ]

Status: RESOLVED FIXED

(Core :: Networking, defect)

Description

[chris hofmann]

very early hours of the 3.6 RC1 release.

looks like a start up crash

http://crash-stats.mozilla.com/report/index/c09e6d4c-0931-4887-89c5-8e8f12100108
Frame  	Module  	Signature [Expand]  	Source
0 	xul.dll 	nsHttpTransaction::DeleteSelfOnConsumerThread 	netwerk/protocol/http/src/nsHttpTransaction.cpp:1081
1 	xul.dll 	nsHttpTransaction::Release 	netwerk/protocol/http/src/nsHttpTransaction.cpp:1112
2 	xul.dll 	nsHttpChannel::SetupTransaction

all reports are Windows NT 5.1.2600 Service Pack 3

and might be the same person submitting.

more reports at
http://crash-stats.mozilla.com/report/list?range_value=2&range_unit=weeks&signature=nsHttpTransaction%3A%3ADeleteSelfOnConsumerThread%28%29&version=Firefox%3A3.6

Comment 1

[timeless]

Signature	nsHttpTransaction::DeleteSelfOnConsumerThread()
UUID	c09e6d4c-0931-4887-89c5-8e8f12100108
Time 	2010-01-08 12:12:08.442359
Uptime	1
Last Crash	11 seconds before submission
Product	Firefox
Version	3.6
Build ID	20100105212446
Branch	1.9.2
OS	Windows NT
OS Version	5.1.2600 Service Pack 3
CPU	x86
CPU Info	GenuineIntel family 15 model 2 stepping 7
Crash Reason	EXCEPTION_ACCESS_VIOLATION
Crash Address	0x0

nsHttpTransaction::Init(PRUint8 caps,
... nsIEventTarget *target,
NS_ASSERTION(target, "ouch");

mConsumerTarget = target;

nsHttpTransaction::DeleteSelfOnConsumerThread()
...
if (NS_FAILED(mConsumerTarget->Dispatch(event, NS_DISPATCH_NORMAL)))


664     mTransaction = new nsHttpTransaction();
676     rv = mTransaction->Init(mCaps, mConnectionInfo, &mRequestHead,
677                             mUploadStream, mUploadStreamHasHeaders,
678                             NS_GetCurrentThread(), callbacks, this,
679                             getter_AddRefs(responseStream));

I don't see any easy way for mConsumerTarget to be null, but the crash sure points to it...

Comment 2

[David Baron :dbaron: (⌚️UTC-5, no longer working on Mozilla)]

This has also (recently) risen up pretty high in 3.6b5 data as well; that makes me think that it's related to some other piece of software that either recently updated or recently became compatible with 3.6.

Histogram of this crash by date, across all Mozilla software:

$ ls | while read FNAME; do echo -n "${FNAME:0:8} "; zcat $FNAME | grep DeleteSelfOnConsumerThread | wc -l; done
20091227 5
20091228 5
20091229 3
20091230 57
20091231 70
20100101 94
20100102 79
20100103 93
20100104 88
20100105 132
20100106 126
20100107 139
20100108 204
20100109 160
20100110 218

Comment 3

[David Baron :dbaron: (⌚️UTC-5, no longer working on Mozilla)]

Attachment: crashes by date and by product/version

Comment 4

[d]

Looking at some crash reports, it seems that an old version (perhaps the latest one too?) of the Yahoo Toolbar could be the cause.

"Yahoo! Toolbar  	{635abd67-4fe9-1b23-4f01-e679fa7484c1}  	1.5.4.20081105  	 1.6.6.20090220"

At least, that's what I find that almost all of the crash reporters have in common. I've seen version 1.5.4 and 1.5.5 in the crashes, not sure if 1.6.6 solves the crashes or if people just haven't upgraded to it yet.

Comment 5

[chris hofmann]

3.6 interesting addons report from 1/18 doesn't show that strong a correlation to the Yahoo Toolbar


  nsHttpTransaction::DeleteSelfOnConsumerThread()|EXCEPTION_ACCESS_VIOLATION (130 crashes)
     81% (105/130) vs.  30% (6735/22797) [e-mail] (Java Quick Starter, http://java.sun.com/javase/downloads/)
     47% (61/130) vs.  33% (7612/22797) {20a82645-c095-46ed-80e3-08825760534b} (Microsoft .NET Framework Assistant, http://www.windowsclient.net/)
     13% (17/130) vs.   0% (17/22797) {a41c2a5a-2e01-9fea-851a-d1d20362797a}
     23% (30/130) vs.  11% (2426/22797) {ABDE892B-13A8-4d1b-88E6-365A6E755758}
     15% (20/130) vs.   4% (824/22797) [e-mail]
      7% (9/130) vs.   0% (67/22797) {FFB96CC1-7EB3-449D-B827-DB661701C6BB}
     30% (39/130) vs.  24% (5373/22797) {CAFEEFAC-0016-0000-0017-ABCDEFFEDCBA}
      9% (12/130) vs.   3% (647/22797) {0b38152b-1b20-484d-a11f-5e04a9b0661f} (Winamp Toolbar, https://addons.mozilla.org/addon/12174)
      7% (9/130) vs.   1% (163/22797) [e-mail] (United States English Dictionary, https://addons.mozilla.org/addon/3497)
     16% (21/130) vs.  10% (2275/22797) {635abd67-4fe9-1b23-4f01-e679fa7484c1} (Yahoo! Toolbar, https://addons.mozilla.org/addon/2032)
      7% (9/130) vs.   2% (409/22797) {CAFEEFAC-0016-0000-0007-ABCDEFFEDCBA} (Java Console, http://java.sun.com/javase/downloads/)

the interesting modules report is equally confusing and widely distributed over a lot of possibilities or combination of possibilities.


     36% (47/130) vs.  24% (5359/22797) mdnsNSP.dll  - Bonjour Namespace Provider\r from Apple Computer, Inc.\

     17% (22/130) vs.   1% (265/22797) SynTPFcs.dll  - component from the software Progressive Touch version 7.5.0 by Synaptics, Inc..

     26% (34/130) vs.  18% (4061/22797) mprapi.dll module that contains functions used to administer Microsoft Windows 2000 routers
     26% (34/130) vs.  18% (4204/22797) adsldpc.dll  ADs LDAP Provider C DLL  Microsoft Corporation 
     26% (34/130) vs.  18% (4078/22797) activeds.dll  Active directory service (router layer)

     22% (29/130) vs.  16% (3706/22797) wzcsapi.dll  - Wireless Zero Configuration service API from Microsof
     22% (29/130) vs.  16% (3761/22797) wmi.dll
     22% (29/130) vs.  17% (3762/22797) wzcsvc.dll
     22% (29/130) vs.  17% (3764/22797) esent.dll
     22% (29/130) vs.  17% (3845/22797) netshell.dll
     22% (29/130) vs.  17% (3889/22797) netman.dll
     22% (29/130) vs.  17% (3944/22797) credui.dll

     21% (27/130) vs.   1% (309/22797) Syncor11.dll - process belonging to the Staccato Systems Synthesizer Driver program
     21% (27/130) vs.  10% (2286/22797) eapolqec.dll  Network Access Protection (NAP) Client Configuration console
     21% (27/130) vs.  10% (2286/22797) qutil.dll
     21% (27/130) vs.  10% (2324/22797) dot3dlg.dll  802.3 UI Helper\r belonging to Microsoft
     21% (27/130) vs.  10% (2334/22797) dot3api.dll
     21% (27/130) vs.  10% (2325/22797) onex.dll
     21% (27/130) vs.  10% (2326/22797) eappprxy.dll  - Microsoft Extensible Authentication Protocol Host
     21% (27/130) vs.  10% (2326/22797) eappcfg.dll

     17% (22/130) vs.   1% (265/22797) SynTPFcs.dll  - component from the software Progressive Touch version 7.5.0 by Synaptics, Inc..

     16% (21/130) vs.   9% (2151/22797) rpmainbrowserrecordplugin.dll  real
     16% (21/130) vs.   9% (2155/22797) nprpffbrowserrecordext.dll

     13% (17/130) vs.   0% (17/22797) jMC2Qj.dll   ???????
     13% (17/130) vs.   0% (23/22797) shellexecutehook.dll
     13% (17/130) vs.   2% (388/22797) vbscript.dll

     10% (13/130) vs.   1% (310/22797) asOEHook.dll - Norton AntiSpam OE Hook\

      7% (9/130) vs.   0% (41/22797) MsgPlusLoader.dll  - msn messenger
      7% (9/130) vs.   1% (148/22797) SASSEH.DLL  - super anti-spyware

Comment 6

[chris hofmann]

we hit a peak of 371 crashes per day on 1/11, and the signature showed up on other releases that day, but at much lower levels.

checking --- 20100111-crashdata.csv DeleteSelfOnConsumerThread
release total-crashes
              DeleteSelfOnConsumerThread crashes
                         pct.
all     223372  371     0.00166091
3.0.15  2063            0
3.0.16  3888            0
3.5.5   5741            0
3.5.6   13189           0
3.5.7   107869  1       9.2705e-06
3.6     15796   370     0.0234237
3.6b5   11261   76      0.00674896
3.6b4   1454    2       0.00137552
3.6b3   572             0
3.6b2   700             0
3.6b1   1917            0

also looks like its happening close to start up and maybe on upgrade or installation for most reporters of the crash

371 total crashes for DeleteSelfOnConsumerThread on 20100111-crashdata.csv
259 start up crashes inside 3 minutes

  93 (no url)
  46 \N (null url)
  45 http://www.mozilla.com/en-US/firefox/3.6b5/whatsnew/

Comment 7

[chris hofmann]

a few of the comments look like its a frequently occurring crash for the people who hit it.   os breakdown is also pretty concentrated.

os breakdown
292     0.787062        Windows NT5.1.2600 Service Pack 3
60      0.161725        Windows NT5.1.2600 Service Pack 2
10      0.0269542       Windows NT5.1.2600
8       0.0215633       Windows NT5.1.2600 Szervizcsomag 3
1       0.00269542      Windows NT5.1.2600 Dodatek Service Pack 3

Comment 8

[chris hofmann]

now moved up to the #8 topcrash on 3.6rc's

Comment 9

[chris hofmann]

some of the low frequency .dll's listed in comment 5 are also found in bug 541293 which is currently on the rise.   wonder if this somehow morf'ed to a different crash.

Comment 10

[chris hofmann]

number #4 top crash in very early 3.6.2 crash data.   maybe connected to migration to new updates.

Comment 11

[Mike Beltzner [:beltzner, not reading bugmail]]

These can't block since we don't know the cause or the fix; we need a better way to say "needs developer investigation and attention." Isn't that what crashkill is for?

Comment 12

[chris hofmann]

re:comment 7.

this signature continues to be 100% win9x and only firefox 3.6+.

might be due to dropped support for win9x in the nspr picked up in firefox 3.6 and current trunk.

Comment 13

[timeless]

Attachment: patch

so, biesi explains:
this also points to the fact that mTransaction->Init failed btw
if it hadn't failed, SetupTransaction wouldn't call DeleteSelfOnConsumerThread

thus, this is simply a local bug.

Comment 14

[Matthew Middleton (:zzxc)]

A user reporting this crash signature on SUMO (with bp-52b6c13b-3cd2-44b3-98df-6b5cf2100720) said that disabling Yahoo Toolbar 1.6.5 stopped the crashing.

(This version of Yahoo Toolbar seems to be from 2008)

Comment 15

[LegNeato]

This would be great to take on the branches once it has landed on trunk FWIW.

Comment 16

[Robert Kaiser]

This signature has jump very significantly yesterday to #18 on 4.0* versions.

Timeless, biesi, is that patch still accurate and should fix the crash? If so, can we please get it reviewed and landed?

Comment 17

[Christian :Biesinger (don't email me, ping me on IRC)]

Comment on attachment 438711 [details] [diff] [review]
patch

ok, looks correct to me. perhaps a comment that mConsumerTarget will be null if Init has not been called or failed would be useful.

Comment 18

[Robert Kaiser]

Thanks, biesi, I hope this can land soon as crashes rising to around #20 for a few days at times are surely something we're happy to go away.

It seems like this is only seen in higher volume in releases, so trunk and aurora don't show crashes from it yet due to their small user sample, 5.0 beta shows very few reports of it already, see https://crash-stats.mozilla.com/report/list?signature=nsHttpTransaction%3A%3ADeleteSelfOnConsumerThread()&version=Firefox%3A5.0 - I guess they'd only rise to significant level when we're shipping to the actual masses.

As the patch/fix looks _really_ simple and we should try to avoid that type of rising crashes on the upcoming releases, nominating for 5 and 6 (I guess approval requests should only come after trunk checkin and baking, right?).

Comment 19

[LegNeato]

Existing crash with known quantity on release and we are at the end of the beta period -> tracking-firefox5: -

Comment 20

[Asa Dotzler [:asa]]

not going to track for Firefox 6, but if you all want to request approval, please do so with an explanation of why it's safe for Aurora.

Comment 21

[Dão Gottwald [:dao]]

http://hg.mozilla.org/mozilla-central/rev/568056abc91e

Comment 22

[Smokey Ardisson (offline for a while; not following bugs - do not email)]

Comment on attachment 438711 [details] [diff] [review]
patch

This crash is our top crash for Camino 2.1--and causes certain users to crash repeatedly at startup--so we'd like approval to take the patch on 1.9.2 as well. 

This bug was originally filed about 1.9.2 crashes, is marked "wanted" for 1.9.2, and has already baked on mozilla-central and various descendants since June.

Comment 23

[Daniel Veditz [:dveditz]]

Comment on attachment 438711 [details] [diff] [review]
patch

Approved for 1.9.2.26, a=dveditz

Comment 24

[Smokey Ardisson (offline for a while; not following bugs - do not email)]

http://hg.mozilla.org/releases/mozilla-1.9.2/rev/6ba95e5c9621

(.26-fixed doesn't exist yet for status1.9.2, or I'd set the flag.)

Comment 25

[u279076]

Unable to verify this fix since I can't reproduce the crash originally as described. Can someone who is able to reproduce this verify the fix? Builds can be found here:
ftp://ftp.mozilla.org/pub/firefox/nightly/3.6.26-candidates/build1/