716391 - Firefox Startup crash @ je_free | xul.dll@0xbb...f | nsAppShell::`. deleting destructor''(unsigned int) with various proxies

Status: RESOLVED WORKSFORME

(Core :: Memory Allocator, defect)

Description

[Marcia Knous [:marcia]]

This bug was filed from the Socorro interface and is 
report bp-1f7886ab-3a2c-45f9-9a59-0f8412120107 .
============================================================= 

Seen while looking at 10.0b3 data. This crash has been present in all of the 10.0 betas, and sits right now at #13 in early B3 data. https://crash-stats.mozilla.com/report/list?signature=je_free%20|%20xul.dll@0xbb999f indicates that are some dupes in the mix so the actual count may be lower.

Will need to look at manual correlations with this one to see what is going on.


Frame 	Module 	Signature [Expand] 	Source
0 	mozutils.dll 	je_free 	memory/jemalloc/jemalloc.c:6497
1 	xul.dll 	xul.dll@0xbb999f 	
2 	xul.dll 	nsAppShell::`vector deleting destructor' 	
3 	xul.dll 	nsBaseAppShell::Release 	js/xpconnect/src/XPCJSID.cpp:320
4 	xul.dll 	nsRefPtr<nsIRunnable>::~nsRefPtr<nsIRunnable> 	obj-firefox/dist/include/nsAutoPtr.h:907
5 	xul.dll 	nsAppStartup::~nsAppStartup 	toolkit/components/startup/nsAppStartup.h:74
6 	xul.dll 	nsAppStartup::Release 	toolkit/components/startup/nsAppStartup.cpp:183
7 	xul.dll 	nsRefPtr<nsIRunnable>::~nsRefPtr<nsIRunnable> 	obj-firefox/dist/include/nsAutoPtr.h:907
8 	xul.dll 	ScopedXPCOMStartup::~ScopedXPCOMStartup 	toolkit/xre/nsAppRunner.cpp:1115
9 	xul.dll 	XRE_main 	
10 	xul.dll 	mozilla::layers::LayerUserDataSet::LayerUserDataSet 	
11 	xul.dll 	XPC_WN_CallMethod 	js/xpconnect/src/XPCWrappedNativeJSOps.cpp:1554
12 	ntdll.dll 	ZwAllocateVirtualMemory 	
13 		@0x21

Comment 1

[Scoobidiver (away)]

It was #64 top crasher in 10.0b2.

I see various uncommon proxy DLLs: asoehook.dll, sahook.dll, snxhk.dll, visicom_antiphishing.dll, idmmkb.dll, ClinckProxy.dll, IGO32.dll, mgAdaptersProxy.dll, TrueTransparencyHook.dll, avsda.dll.
In some reports, there's no uncommon DLL.

Comment 2

[Sheila Mooney]

Sitting at #13 on b3 which is much higher than b2.

Comment 3

[Sheila Mooney]

I am going to track this for 10.

Comment 4

[Alex Keybl [:akeybl]]

CC'ing Kyle Huey in case this is somehow related to bug 414946 (should have been Mac-only though), Josh Aas to follow along in case this does end up being proxy/networking related, and Anthony/Juan to see if they can line up some testing with the software associated with DLLs in comment 1.

Comment 5

[u279076]

Did some Googling to track down software (versions unknown):

asoehook.dll => Norton Internet Security
sahook.dll => McAfee Site Advisor
snxhk.dll => Avast! AntiVirus / Internet Security
visicom_antiphishing.dll => Visicom Media Anti-Phishing Domain Advisor (Powered by Panda Security)
idmmkb.dll => Internet Download Manager by Tonec Inc.
clinckproxy.dll => Ideacts Innovations Pvt. Ltd.
IGO32.dll => Origin IGO by Electronic Arts
mgAdaptersProxy.dll => SweetIM for Messenger by SweetIM Technologies Ltd.
TrueTransparencyHook.dll => TrueTransparency by Lefreut Software
avsda.dll => Avira Antivirus Premium by Avira Operations GmbH & Co. KG

What's the next steps? How can we isolate and track down specific versions? Is there a dll that is reported more than others?

Comment 6

[Marcia Knous [:marcia]]

Per triage meeting today, we will be getting correlation reports for this via Socorro.

Comment 7

[Marcia Knous [:marcia]]

Requested a correlation report for this signature in Bug 717041.

Comment 8

[Scoobidiver (away)]

Here are the requested correlations:
  je_free | xul.dll@0xbb999f|EXCEPTION_ACCESS_VIOLATION_READ (46 crashes)
     96% (44/46) vs.  44% (8682/19911) hnetcfg.dll
     96% (44/46) vs.  44% (8728/19911) wshtcpip.dll
     96% (44/46) vs.  46% (9140/19911) comres.dll
     96% (44/46) vs.  46% (9205/19911) ws2help.dll
     96% (44/46) vs.  53% (10478/19911) mpr.dll
     78% (36/46) vs.  59% (11691/19911) secur32.dll
     17% (8/46) vs.   0% (16/19911) prio.dll
    100% (46/46) vs.  84% (16742/19911) feclient.dll
     17% (8/46) vs.   5% (1038/19911) mgAdaptersProxy.dll
    100% (46/46) vs.  89% (17652/19911) browsercomps.dll
    100% (46/46) vs.  89% (17669/19911) softokn3.dll
    100% (46/46) vs.  89% (17672/19911) firefox.exe
    100% (46/46) vs.  89% (17678/19911) xpcom.dll
    100% (46/46) vs.  89% (17735/19911) dbghelp.dll
      7% (3/46) vs.   1% (121/19911) LVPrcInj.dll
    100% (46/46) vs.  94% (18753/19911) mswsock.dll

Comment 9

[Scoobidiver (away)]

The xul address shifted in 10.0b4.

>      17% (8/46) vs.   0% (16/19911) prio.dll
Process Priority Saver (see http://www.prnwatch.com/prio.html)
>      17% (8/46) vs.   5% (1038/19911) mgAdaptersProxy.dll
SweetIM for Messenger by SweetIM Technologies Ltd (see http://sweetim.com/?mlc=EN-US&)

Comment 10

[Scoobidiver (away)]

As it's a startup crash, correlations change across days. The latest ones give:
  je_free | xul.dll@0xbbc99f | nsAppShell::`vector deleting destructor''(unsigned int)|EXCEPTION_ACCESS_VIOLATION_READ (67 crashes)
     99% (66/67) vs.  57% (7652/13378) lpk.dll (Windows Language Pack)
     49% (33/67) vs.  15% (2023/13378) snxhk.dll (Avast! AntiVirus / Internet Security)
     33% (22/67) vs.   1% (189/13378) guard32.dll (Comodo Firewall / Internet Security)
      6% (4/67) vs.   0% (11/13378) prio.dll (Process Priority Saver)

Comment 11

[Scoobidiver (away)]

Here are 10.0.1 correlations per module over consecutive days:
* Feb 13:
     27% (32/118) vs.  14% (3564/25105) snxhk.dll (Avast! AntiVirus)
     25% (29/118) vs.  12% (3043/25105) aswJsFlt.dll (Avast! AntiVirus)
* Feb 14:
     32% (32/101) vs.  13% (5933/44591) snxhk.dll (Avast! AntiVirus)
     26% (26/101) vs.  11% (5037/44591) aswJsFlt.dll (Avast! AntiVirus)
* Feb 15:
     19% (14/75) vs.   5% (2482/49762) datamngr.dll (SearcQu Toolbar)
      8%  (6/75) vs.   1% (568/49762) Iminent.WinCore.dll (Iminent IMBooster)
      9% (7/75) vs.   4% (1769/49762) mgAdaptersProxy.dll (SweetIM)
* Feb 16:
No data
* Feb 17:
      6% (5/82) vs.   0% (148/54820) prio.dll (Prio - Priority Saver)

It would be interesting to have correlations per LSP.

Comment 12

[Alex Keybl [:akeybl]]

(In reply to Anthony Hughes, Mozilla QA (irc: ashughes) from comment #5)
> What's the next steps? How can we isolate and track down specific versions?
> Is there a dll that is reported more than others?

Based upon Scoobidiver's correlation report, let's try reproducing this startup crash with Avast! AntiVirus.

Comment 13

[u279076]

Currently pulling down an XP VM to test this -- if someone else is already set up please jump in. It will take me over an hour to pull the VM.

Comment 14

[Sheila Mooney]

Anthony, any luck reproducing this?

Comment 15

[u279076]

(In reply to Sheila Mooney from comment #14)
> Anthony, any luck reproducing this?

No, not yet. :/

Comment 16

[u279076]

I've tried testing this and so far have yet to crash. My basic methodology was to install a previous version of Firefox (tried 10.0, 10.0.1, 10.0b3, 10.0b5) then install Avast! Antivirus free then browse around for an hour or so, update to Firefox 10.0.2 and continue to browse. I also peppered a few restarts in there. I even tried rapidly quitting and restarting Firefox several times. All to no success -- I can't get Firefox 10.* to crash with Avast! installed.

One thing to note is that when installing Avast! it installs an "avast! WebRep" add-on. It installs version 6.0.1367 which is apparently new as of today.

I supposes it's possible that they updated the add-on and it fixed this crash though I don't know how likely that is.

I'm no expert in crash investigation, and unfortunately Marcia is away. I'd be happy to help further given the appropriate guidance.

Sorry I couldn't be of more help.

Comment 17

[Scoobidiver (away)]

(In reply to Anthony Hughes, Mozilla QA (irc: ashughes) from comment #16) 
> I supposes it's possible that they updated the add-on and it fixed this
> crash though I don't know how likely that is.
It's not the case. See today's correlation per module for 10.0.2:
     19% (18/95) vs.  13% (7199/56926) snxhk.dll
          1% (1/95) vs.   0% (16/56926) 5.1.864.0
          1% (1/95) vs.   0% (187/56926) 6.0.1203.0
          1% (1/95) vs.   1% (811/56926) 6.0.1289.0
         16% (15/95) vs.  10% (5502/56926) 6.0.1367.0

FWIW, I have used Avast! AV for years without the Avast! WebRep add-on and haven't crashed with those signatures.
The add-on installation in browsers is managed from the Avast! AV UI. It must be noted that there's a Check Automatically New Browser Installation setting related to this add-on.

Comment 18

[Marcia Knous [:marcia]]

I tried reproducing by installing Avast Internet suite as well as the Easypass toolbar but so far no luck yet. I installed version 6.0.1367 of avast! WebRep. I will give this another attempt tomorrow.

Comment 19

[Alex Keybl [:akeybl]]

I went through the crash comments again but saw no new information. This crash has dropped significantly since we've released FF10, but that could be for any number of reasons.

Is there any way to dive deeper into the data and find the first date this crash was reported? It's either a regression in Firefox or something we tickled in third party software considering 100% of the crashes are on FF10.

Comment 20

[Marcia Knous [:marcia]]

I am investigating some malware that I saw in a few reports that had more uptime. Will report back.

Comment 21

[Alex Keybl [:akeybl]]

(In reply to Marcia Knous [:marcia] from comment #20)
> I am investigating some malware that I saw in a few reports that had more
> uptime. Will report back.

Do we still believe malware to be the major cause of this startup crasher?

Comment 22

[Marcia Knous [:marcia]]

I don't have any solid evidence yet that malware is the cause. What I do know is this crash was likely around in 9.0.1, but with a slightly different signature: https://crash-stats.mozilla.com/report/list?signature=je_free%20|%20xul.dll@0x2eba9f%20|%20clearPrefEntry. I am going to look at some more manual correlations now - the ones from the today did not show much.

Comment 23

[Scoobidiver (away)]

(In reply to Marcia Knous [:marcia] from comment #22)
> https://crash-stats.mozilla.com/report/list?signature=je_free%20|%20xul.
> dll@0x2eba9f%20|%20clearPrefEntry.
This one is bug 677082 with a different stack.

Comment 24

[Marcia Knous [:marcia]]

Latest manual correlations for largest volume signature still show a correlation to Avast:

je_free | xul.dll@0xbbc98f | nsAppShell::`scalar deleting destructor''(unsigned int)|EXCEPTION_ACCESS_VIOLATION_READ (86 crashes)
     99% (85/86) vs.  84% (61181/72500) feclient.dll
     19% (16/86) vs.  11% (8003/72500) aswJsFlt.dll
     98% (84/86) vs.  91% (65689/72500) browsercomps.dll
     19% (16/86) vs.  13% (9322/72500) snxhk.dll
    100% (86/86) vs.  94% (68486/72500) firefox.exe
    100% (86/86) vs.  94% (68503/72500) xpcom.dll

avast! Script Blocking filter library is the aswJsFlt.dll according to a Google search.

Comment 25

[Scoobidiver (away)]

It only affects 10.0 ESR.

Comment 26

[u279076]

(In reply to Scoobidiver from comment #25)
> It only affects 10.0 ESR.

Does it affect all 10.0 ESR versions (ie. up to and including 10.0.5esr) or only 10.0-final ESR?

Comment 27

[Scoobidiver (away)]

There are 3 crashes from the same user in 10.0.5 ESR: https://crash-stats.mozilla.com/report/list?product=Firefox&version=Firefox%3A10.0.5esr&query_search=signature&query_type=contains&query=nsAppShell%3A%3A%60&reason_type=contains&do_query=1&signature=je_free%20|%20xul.dll%400xbbc99f%20|%20nsAppShell%3A%3A%60scalar%20deleting%20destructor%27%27%28unsigned%20int%29

Comment 28

[u279076]

Looking at the last 4 weeks, there are 21 crashes, all against 10.0.5esr:
https://crash-stats.mozilla.com/report/list?product=Firefox&query_search=signature&query_type=exact&query=je_free%20|%20xul.dll%400xbbc99f%20|%20nsAppShell%3A%3A%60scalar%20deleting%20destructor%27%27%28unsigned%20int%29&reason_type=contains&date=06%2F22%2F2012%2016%3A31%3A59&range_value=4&range_unit=weeks&hang_type=any&process_type=any&do_query=1&signature=je_free%20|%20xul.dll%400xbbc99f%20|%20nsAppShell%3A%3A%60scalar%20deleting%20destructor%27%27%28unsigned%20int%29

Considering Firefox 10.0.5esr came out 3 weeks ago, does this indicate a possible regression in Firefox 10.0.5esr?

Comment 29

[Scoobidiver (away)]

This bug is a regression in 10.0 so it's not surprising that it has appeared in all 10.0 ESR builds.

Comment 30

[u279076]

I don't see that it appeared on ALL 10.0esr builds, only 10.0.5esr builds. Where are you seeing otherwise?

Comment 31

[Scoobidiver (away)]

(In reply to Anthony Hughes, Mozilla QA (irc: ashughes) from comment #30)
> I don't see that it appeared on ALL 10.0esr builds, only 10.0.5esr builds.
> Where are you seeing otherwise?
See https://crash-stats.mozilla.com/query/query?product=Firefox&version=Firefox%3A10.0.6esrpre&version=Firefox%3A10.0.5esr&version=Firefox%3A10.0.4esr&version=Firefox%3A10.0.3esr&version=Firefox%3A10.0.2esr&range_value=4&range_unit=weeks&query_search=signature&query_type=contains&query=je_free+|+xul.dll%400xbb&do_query=1

Comment 32

[u279076]

I tried to reproduce this again but failed. Can we do some outreach to the Enterprise mailing list?

Comment 33

[Scoobidiver (away)]

(In reply to Anthony Hughes, Mozilla QA (irc: ashughes) from comment #32)
> Can we do some outreach to the Enterprise mailing list?
Only the top-10 ESR crashers can be qualified for a fix search. This one is only #289 in 10.0.5 ESR and fixed in 17.0 ESR, so it would be a waste of time.

Comment 34

[u279076]

Given recent comments, removing qawanted. Please re-add if other leads come to light which QA can investigate.

Comment 35

[Virtual_ManPL [:Virtual] 🇵🇱 - (please needinfo? me - so I will see your comment/reply/question/etc.)]

I'm marking this bug as WORKSFORME as bug crashlog signature didn't appear from a long time (over half year) in Firefox (except some obsolete Fx <11).