Closed Bug 663466 Opened 11 years ago Closed 7 months ago

crash @ __RtlUserThreadStart | _RtlUserThreadStart

Categories

(Firefox :: General, defect)

x86
Windows 7
defect
Not set
critical

Tracking

()

RESOLVED WORKSFORME
Tracking Status
firefox21 --- wontfix
firefox22 + wontfix
firefox23 - affected

People

(Reporter: marcia, Unassigned)

References

Details

(Keywords: crash, topcrash-win, Whiteboard: [startupcrash])

Crash Data

This bug was filed from the Socorro interface and is 
report bp-40656464-c36a-409f-a8eb-193ac2110608 .
============================================================= 

Seen while reviewing crash stats. https://crash-stats.mozilla.com/report/list?signature=__RtlUserThreadStart to the crashes which span all versions.

Some of the comments mention having trouble with toolbars and a fair number have the addon compatibility extension installed. Addon correlation:

78% (39/50) vs.   1% (364/27164) compatibility@addons.mozilla.org
52% (26/50) vs.   0% (26/27164) {4D1E692F-D179-413b-A987-EEEAAD85DDB3}
52% (26/50) vs.   0% (26/27164) pbupload@photobucket.com (Photobucket Uploader, https://addons.mozilla.org/addon/10035)
52% (26/50) vs.   0% (27/27164) plugin2@gameplaylabs.com
52% (26/50) vs.   0% (28/27164) gamesbar@oberon-media.com
52% (26/50) vs.   0% (28/27164) {f92a9fe4-2850-4198-b9d5-279880e49b16}
52% (26/50) vs.   0% (32/27164) searchtoolbar@zugo.com
52% (26/50) vs.   0% (42/27164) {8A9386B4-E958-4c4c-ADF4-8F26DB3E4829} (PriceGong, https://addons.mozilla.org/addon/10433)
52% (26/50) vs.   1% (152/27164) {1E73965B-8B48-48be-9C8D-68B920ABC1C4}
52% (26/50) vs.   1% (302/27164) {635abd67-4fe9-1b23-4f01-e679fa7484c1} (Yahoo! Toolbar, https://addons.mozilla.org/addon/2032)
52% (26/50) vs.   1% (389/27164) avg@igeared
52% (26/50) vs.   2% (524/27164) {CAFEEFAC-0016-0000-0026-ABCDEFFEDCBA}
52% (26/50) vs.   2% (641/27164) engine@conduit.com
36% (18/50) vs.   0% (21/27164) {32a1fd71-835e-4b11-8e54-886fda0b4c89}
36% (18/50) vs.   3% (907/27164) personas@christopher.beard (Personas, https://addons.mozilla.org/addon/10900)
52% (26/50) vs.  22% (6000/27164) {CAFEEFAC-0016-0000-0024-ABCDEFFEDCBA}
26% (13/50) vs.   0% (14/27164) langpack-ru@firefox.mozilla.org
26% (13/50) vs.   0% (15/27164) sharing@addons.mozilla.org (Add-on Collector, https://addons.mozilla.org/addon/11950)
26% (13/50) vs.   0% (20/27164) MafiaaFire@mafiaafire.com
26% (13/50) vs.   0% (32/27164) {cd617375-6743-4ee8-bac4-fbf10f35729e} (RightToClick, https://addons.mozilla.org/addon/12572)
26% (13/50) vs.   0% (35/27164) ru@dictionaries.addons.mozilla.org (Russian spellchecking dictionary, https://addons.mozilla.org/addon/3703)
26% (13/50) vs.   0% (41/27164) fastdial@telega.phpnet.us (Fast Dial, https://addons.mozilla.org/addon/5721)
26% (13/50) vs.   0% (46/27164) helper@savefrom.net
26% (13/50) vs.   0% (63/27164) SkipScreen@SkipScreen (SkipScreen, https://addons.mozilla.org/addon/11243)
26% (13/50) vs.   2% (466/27164) {19503e42-ca3c-4c27-b1e2-9cdb2170ee34} (FlashGot, https://addons.mozilla.org/addon/220)
26% (13/50) vs.   3% (887/27164) {D4DD63FA-01E4-46a7-B6B1-EDAB7D6AD389} (Download Statusbar, https://addons.mozilla.org/addon/26)
32% (16/50) vs.  10% (2777/27164) {d10d0bf8-f5b5-c8b4-a8b2-2b9879e08c5d} (Adblock Plus, https://addons.mozilla.org/addon/1865)
26% (13/50) vs.   6% (1623/27164) {CAFEEFAC-0016-0000-0025-ABCDEFFEDCBA}
12% (6/50) vs.   0% (32/27164) {37964A3C-4EE8-47b1-8321-34DE2C39BA4D}
92% (46/50) vs.  82% (22208/27164) testpilot@labs.mozilla.com (Mozilla Labs - Test Pilot, https://addons.mozilla.org/addon/13661)
10% (5/50) vs.   0% (5/27164) {d21e1d10-117c-11df-8a39-0800200c9a66}
10% (5/50) vs.   0% (8/27164) {6236BA26-C117-4007-928C-DE0716C7FA78} (Chameleon Bob: layouts for Facebook, https://addons.mozilla.org/addon/11584)
10% (5/50) vs.   0% (9/27164) {6236BA26-C117-4007-928C-DE0716C7FA68} (Express Tab, https://addons.mozilla.org/addon/14584)
10% (5/50) vs.   0% (14/27164) {6236BA26-C117-4007-928C-DE0716C7FA48}
10% (5/50) vs.   0% (24/27164) {6236BA26-C117-4007-928C-DE0716C7FA38}
10% (5/50) vs.   0% (92/27164) {3e9a3920-1b27-11da-8cd6-0800200c9a66}
14% (7/50) vs.   8% (2174/27164) {CAFEEFAC-0016-0000-0023-ABCDEFFEDCBA}

https://crash-stats.mozilla.com/report/index/40656464-c36a-409f-a8eb-193ac2110608

Frame 	Module 	Signature [Expand] 	Source
0 		@0x3f90000 	
1 	ntdll.dll 	__RtlUserThreadStart 	
2 	ntdll.dll 	_RtlUserThreadStart
Crash Signature: [@ __RtlUserThreadStart]
Appears at the #26 spot on 8.0 and #18 on 8.0.1 over the past week.
Keywords: topcrash
It's in the long list of bugs correlated to Спутник @Mail.Ru and Yandex.Bar:
68% (101/148) vs.   7% (3725/51702) yasearch@yandex.ru (Yandex.Bar, https://addons.mozilla.org/addon/3495)
64% (94/148) vs.   6% (3010/51702) {37964A3C-4EE8-47b1-8321-34DE2C39BA4D}
Around 92% of these are occurring < 1 min.
Whiteboard: startupcrash
Depends on: 720655
Crash Signature: [@ __RtlUserThreadStart] → [@ __RtlUserThreadStart] [@ __RtlUserThreadStart | _RtlUserThreadStart]
Summary: crash [@ __RtlUserThreadStart] → crash @ __RtlUserThreadStart | _RtlUserThreadStart
Crash Signature: [@ __RtlUserThreadStart] [@ __RtlUserThreadStart | _RtlUserThreadStart] → [@ __RtlUserThreadStart] [@ __RtlUserThreadStart | _RtlUserThreadStart ]
This seems to have gone way down in volume. Sitting at #63 on Fx12. Removing the top crash keyword.
No longer depends on: 720655
Keywords: topcrash
Depends on: 720655
It's #27 browser crasher in 21.0, #14 in 22.0b3 and 23.0a2, and #40 in 24.0a1.

Windows XP is not affected. 93% of crashes are at startup.

Here are interesting correlations:
* 21.0:
  __RtlUserThreadStart | _RtlUserThreadStart|EXCEPTION_ACCESS_VIOLATION_EXEC (270 crashes)
    64% (173/270) vs.  12% (13495/111807) aswJsFlt.dll (Avast! Antivirus Script Blocking filter library)
          3% (8/270) vs.   0% (398/111807) 6.0.1367.0
          0% (1/270) vs.   0% (23/111807) 7.0.1451.402
          1% (4/270) vs.   0% (188/111807) 7.0.1455.186
          1% (3/270) vs.   0% (369/111807) 7.0.1456.418
          2% (6/270) vs.   1% (846/111807) 7.0.1466.549
         15% (40/270) vs.   2% (2717/111807) 7.0.1474.765
         17% (46/270) vs.   3% (3571/111807) 8.0.1483.72
          1% (4/270) vs.   0% (333/111807) 8.0.1488.286
         22% (60/270) vs.   4% (4040/111807) 8.0.1489.300
    66% (177/270) vs.  17% (18596/111807) snxhk.dll (Avast! Antivirus)
          0% (1/270) vs.   0% (125/111807) 6.0.1125.0
          3% (8/270) vs.   0% (415/111807) 6.0.1367.0
          0% (1/270) vs.   0% (162/111807) 6.0.999.0
          0% (1/270) vs.   0% (25/111807) 7.0.1451.402
          1% (4/270) vs.   0% (209/111807) 7.0.1455.186
          1% (3/270) vs.   0% (384/111807) 7.0.1456.418
          2% (6/270) vs.   1% (887/111807) 7.0.1466.549
         15% (40/270) vs.   3% (2851/111807) 7.0.1474.765
          0% (1/270) vs.   0% (339/111807) 8.0.1482.45
         17% (46/270) vs.   5% (5265/111807) 8.0.1483.72
          1% (4/270) vs.   0% (484/111807) 8.0.1488.286
         23% (62/270) vs.   6% (6378/111807) 8.0.1489.300
     52% (141/270) vs.   8% (8536/111807) BrowserProtect.dll (PerformerSoft Protector)
          1% (3/270) vs.   1% (752/111807) 2.6.1095.52
         51% (138/270) vs.   6% (7065/111807) 2.6.1249.132
     24% (66/270) vs.   2% (2608/111807) Iminent.WinCore.dll (Iminent)
          1% (2/270) vs.   0% (178/111807) 3.47.0.0
          4% (12/270) vs.   0% (89/111807) 5.18.52.0
          0% (1/270) vs.   0% (44/111807) 5.26.21.0
          0% (1/270) vs.   0% (64/111807) 5.29.41.0
          0% (1/270) vs.   0% (74/111807) 5.35.51.0
          1% (2/270) vs.   0% (22/111807) 5.48.42.0
          1% (2/270) vs.   0% (47/111807) 5.51.31.0
          0% (1/270) vs.   0% (96/111807) 5.52.31.0
          2% (5/270) vs.   0% (325/111807) 6.14.22.0
          1% (4/270) vs.   0% (234/111807) 6.17.41.0
         12% (32/270) vs.   0% (225/111807) 6.21.22.0
          1% (3/270) vs.   1% (708/111807) 6.4.56.0
     18% (48/270) vs.   4% (4215/111807) mgAdaptersProxy.dll (SweetIM Messenger)
          1% (3/270) vs.   0% (99/111807) 3.3.0.7
          0% (1/270) vs.   0% (58/111807) 3.4.0.5
          1% (2/270) vs.   0% (35/111807) 3.5.0.8
          1% (4/270) vs.   0% (301/111807) 3.6.0.2
          5% (13/270) vs.   0% (296/111807) 3.6.0.8
          2% (6/270) vs.   1% (771/111807) 3.7.0.5
          7% (19/270) vs.   2% (2304/111807) 3.7.0.7

* 22.0 Beta:
  __RtlUserThreadStart | _RtlUserThreadStart|EXCEPTION_ACCESS_VIOLATION_WRITE (323 crashes)
     33% (106/323) vs.   1% (515/37606) datamngr.dll (Bandoo's Data Manager)
         33% (106/323) vs.   1% (478/37606) 1.0.0.1
     14% (44/323) vs.   1% (326/37606) McSACorePS.dll (McAfee SiteAdvisor)
         14% (44/323) vs.   1% (260/37606) 3.6.0.137
          0% (0/323) vs.   0% (3/37606) 3.6.1.106
     14% (44/323) vs.   1% (329/37606) saplugin.dll (McAfee SiteAdvisor)
         14% (44/323) vs.   1% (267/37606) 3.6.0.187
     14% (44/323) vs.   2% (759/37606) sahook.dll (McAfee SiteAdvisor)
         14% (44/323) vs.   1% (489/37606) 3.6.0.187
          0% (0/323) vs.   0% (155/37606) 3.6.1.193

* 23.0a2:
  __RtlUserThreadStart | _RtlUserThreadStart|EXCEPTION_ACCESS_VIOLATION_WRITE (25 crashes)
     44% (11/25) vs.   2% (37/2095) avsda.dll (Avira Free Antivirus)
         44% (11/25) vs.   1% (28/2095) 13.6.2.666
     24% (6/25) vs.   0% (6/2095) KainyX.dll

I don't experience a startup crash with Avast! Antivirus Basic Edition with or without the extension in 21.0.

More reports at:
https://crash-stats.mozilla.com/report/list?signature=__RtlUserThreadStart+|+_RtlUserThreadStart
Keywords: topcrash
OS: Windows NT → Windows 7
Whiteboard: startupcrash → [startupcrash]
Version: unspecified → Trunk
We have a strong suspicion that this is a 3rd party issue since this wasn't a top crash while FF22 was on Beta.

One thing we saw in the crash comments that really concerned us was:

"any ideas, each time i open it i see the open in safe mode prompt for a split second then it goes straight to crash report and restarting firefox or quiting firefox doesnt seem to do anything"
Flags: needinfo?(kairo)
That needinfo was meant to ask whether Kairo could check how many unique users are affected on each channel.
Including MattN, since he worked on bug 294260, and may be able to shine some light onto what situations would warrant the crash comment from comment 6. Especially since we think this was tickled by a third party.
(In reply to Alex Keybl [:akeybl] from comment #6)
> One thing we saw in the crash comments that really concerned us was:
> 
> "any ideas, each time i open it i see the open in safe mode prompt for a
> split second then it goes straight to crash report and restarting firefox or
> quiting firefox doesnt seem to do anything"

I suspect it's just the startup crash detection doing its job but we are still hitting the crash in safe mode while the dialog is open. This elimates plugins as the cause because they shouldn't be started that early.
(In reply to Alex Keybl [:akeybl] from comment #7)
> That needinfo was meant to ask whether Kairo could check how many unique
> users are affected on each channel.

breakpad=> SELECT version,COUNT(*) as crashes,COUNT(DISTINCT client_crash_date - install_age  * interval '1 second') as installations FROM reports WHERE product='Firefox' AND signature='__RtlUserThreadStart | _RtlUserThreadStart' AND date_processed BETWEEN '2013-05-29' AND '2013-06-05' GROUP BY version;
  version  | crashes | installations 
-----------+---------+---------------
[...]
 21.0      |    2796 |          2014
 22.0      |    1184 |           262
 23.0a2    |      79 |            21
 24.0a1    |      21 |             7
[...]

Starting next week, we should get those numbers in Signature Summaries on Socorro as well.
Flags: needinfo?(kairo)
Gavin - any suggestion on who could look at this startup crash? I believe Matt's comment 9 suggests this is an issue that's likely completely on our side.
Assignee: nobody → gavin.sharp
(In reply to Alex Keybl [:akeybl] from comment #11)
> I believe Matt's comment 9 suggests this is an issue that's likely completely on our
> side.

He said "not plugins", which doesn't necessarily imply "completely on our side" :)

I don't have any ideas here, I think bsmedberg is a better wild-guess owner than I am.
Assignee: gavin.sharp → benjamin
What's the current rank of this? It's unlikely that I'll be able to work on this for a couple weeks at least.

Most of these appear to be Flash-process crashes that are not labeled as such. See e.g. https://crash-stats.mozilla.com/report/index/53347fea-bcf4-4e10-b87f-aa1062130614

It would be helpful if somebody could classify these by process by searching the module list for firefox.exe or plugin-container.exe or FlashPlayerPlugin*.exe
Flags: needinfo?
A comment in French says that Firefox crashes (at startup) after a Skype update.
Most comments are in Portuguese.

(In reply to Benjamin Smedberg  [:bsmedberg] from comment #13)
> What's the current rank of this?
It's #15 top browser crasher in 21.0, #17 in 22.0b5 and 23.0a2. It jumps six ranks in one week.
For the last day, it's even #10 top crasher in 21.0.

> Most of these appear to be Flash-process crashes that are not labeled as
> such. See e.g.
> https://crash-stats.mozilla.com/report/index/53347fea-bcf4-4e10-b87f-
> aa1062130614
This one is bug 852096 and doesn't have the same correlations as this one. Compare comment 5 and bug 852096.

> It would be helpful if somebody could classify these by process by searching
> the module list for firefox.exe or plugin-container.exe or
> FlashPlayerPlugin*.exe
Here they are for 21.0:
  __RtlUserThreadStart | _RtlUserThreadStart|EXCEPTION_ACCESS_VIOLATION_EXEC (602 crashes)
    100% (601/602) vs.  69% (69637/101515) firefox.exe
No plugin-container.exe or FlashPlayerPlugin_*.exe correlations.
Flags: needinfo?
(In reply to Scoobidiver from comment #14)
> A comment in French says that Firefox crashes (at startup) after a Skype
> update.
Skype 6.5 was released on June 5 (second spike) and 6.3 on May 13 (first spike in comment 5). See http://blogs.skype.com/category/garage-updates/
It's #4 top browser crasher in the first days of 22.0 and still correlated to Avast Antivirus.
connor, can you contact Avast and see if they have any clues on this one? There's not much useful given the data we have other than the correlation. It looks like somebody is injecting code and trying to launch a thread but failing somehow.
Flags: needinfo?(mconnor)
#15 on 23.0b1 and going down so untracking.
#8 in Fx 23.0 Release
Current correlations:

Addons:
  __RtlUserThreadStart | _RtlUserThreadStart|EXCEPTION_ACCESS_VIOLATION_EXEC (425 crashes)
     66% (280/425) vs.   1% (1257/95359) {1B33E42F-EF14-4cd3-B6DC-174571C4349C}
     43% (181/425) vs.   1% (1298/95359) cehomepage@mozillaonline.com
     42% (179/425) vs.   1% (1320/95359) cpmanager@mozillaonline.com
     41% (175/425) vs.   1% (1185/95359) cemigration@mozillaonline.com
     39% (165/425) vs.   1% (1264/95359) tabimprovelite@mozillaonline.com
     39% (164/425) vs.   1% (1228/95359) quicklaunch@mozillaonline.com
     37% (157/425) vs.   1% (1044/95359) coba@mozilla.com.cn
     36% (155/425) vs.   1% (1103/95359) semodifier@mozillaonline.com
     35% (150/425) vs.   1% (912/95359) commonfix@mozillaonline.com
     33% (142/425) vs.   1% (843/95359) account@mozillaonline.com
     33% (140/425) vs.   1% (855/95359) easyscreenshot@mozillaonline.com
     31% (132/425) vs.   1% (1024/95359) share_all_cn@mozillaonline.com
     14% (60/425) vs.   0% (321/95359) xthunder@lshai.com
     12% (49/425) vs.   1% (567/95359) livemargins@mozillaonline.com
      9% (38/425) vs.   0% (185/95359) {3ec6f6a6-94e7-41d5-8c36-cec334456e1e}
      7% (30/425) vs.   0% (242/95359) fontsetter@mozillaonline.com
      7% (30/425) vs.   1% (947/95359) personas@christopher.beard (Personas, https://addons.mozilla.org/addon/10900)
      6% (26/425) vs.   0% (218/95359) zoompanel@mozillaonline.com
      6% (25/425) vs.   0% (334/95359) netvideohunter@netvideohunter.com (NetVideoHunter Video Downloader, https://addons.mozilla.org/addon/7447)

Modules:
  __RtlUserThreadStart | _RtlUserThreadStart|EXCEPTION_ACCESS_VIOLATION_EXEC (425 crashes)
     99% (419/425) vs.  42% (39881/95359) DWrite.dll
     96% (410/425) vs.  40% (38120/95359) cryptsp.dll
     87% (370/425) vs.  39% (37657/95359) nlaapi.dll
     87% (370/425) vs.  40% (37672/95359) NapiNSP.dll
     99% (422/425) vs.  52% (49808/95359) ntmarta.dll
     82% (348/425) vs.  35% (33544/95359) explorerframe.dll
     72% (305/425) vs.  25% (23918/95359) d2d1.dll
     82% (348/425) vs.  35% (33568/95359) dui70.dll
     86% (366/425) vs.  39% (37618/95359) pnrpnsp.dll
     73% (311/425) vs.  27% (25303/95359) dxgi.dll
     83% (354/425) vs.  37% (35254/95359) FWPUCLNT.DLL
     82% (348/425) vs.  36% (34383/95359) duser.dll
     66% (280/425) vs.  25% (23899/95359) d3d10_1core.dll
     66% (280/425) vs.  25% (23899/95359) d3d10_1.dll
     98% (415/425) vs.  58% (55059/95359) samcli.dll
     98% (415/425) vs.  58% (55082/95359) wkscli.dll
     98% (415/425) vs.  58% (55085/95359) netutils.dll
     98% (415/425) vs.  58% (55112/95359) srvcli.dll
    100% (425/425) vs.  62% (58722/95359) propsys.dll
     96% (410/425) vs.  59% (56296/95359) profapi.dll
     38% (163/425) vs.   1% (1085/95359) oledlg.dll
     58% (246/425) vs.  21% (19746/95359) winhttp.dll
     38% (162/425) vs.   1% (930/95359) MinHook.32.dll
     91% (385/425) vs.  54% (51065/95359) sspicli.dll
     37% (159/425) vs.   1% (579/95359) msimtf.dll
     98% (415/425) vs.  61% (58234/95359) devobj.dll
     37% (159/425) vs.   1% (892/95359) npcoba32.dll
     98% (415/425) vs.  61% (58352/95359) KERNELBASE.dll
     98% (415/425) vs.  61% (58353/95359) sechost.dll
     98% (415/425) vs.  61% (58353/95359) CRYPTBASE.dll
     53% (224/425) vs.  17% (15762/95359) oleacc.dll
     39% (166/425) vs.   3% (3194/95359) mshtml.dll
     98% (415/425) vs.  62% (59085/95359) cfgmgr32.dll
    100% (425/425) vs.  65% (61825/95359) winnsi.dll
    100% (425/425) vs.  65% (61843/95359) IPHLPAPI.DLL


Looking at the add-ons, this looks very heavily correlated with the Mozilla China (mozillaonline.com) add-ons. It may be caused by something they ship there or by some other Chinese thing. We need input from them. Release Management, do you know who to ask there?
Flags: needinfo?(release-mgmt)
Let's ask Jorge first since he's more familiar with addons community
Flags: needinfo?(release-mgmt) → needinfo?(jorge)
{1B33E42F-EF14-4cd3-B6DC-174571C4349C} and xthunder@lshai.com are also Chinese add-ons, but not developed by Mozilla Online, so it's possible that this crash has more to do with the Chinese build of Firefox than the actual add-ons.

Hong, can you please look into this?
Flags: needinfo?(jorge) → needinfo?(htang)
The crash here looks a little similar to bug 902790, and the symptoms might also match (ie, that Avast or something is preventing the plugin child process from starting)?
The "or something" is the big question.
this signature drifts in and out of topcrash status

topcrash is being replaced by more precise keywords per https://bugzilla.mozilla.org/show_bug.cgi?id=927557#c3
Keywords: topcrashtopcrash-win
Depends on: 948614
Assignee: benjamin → nobody
(In reply to Jorge Villalobos [:jorgev] from comment #22)
> {1B33E42F-EF14-4cd3-B6DC-174571C4349C} and xthunder@lshai.com are also
> Chinese add-ons, but not developed by Mozilla Online, so it's possible that
> this crash has more to do with the Chinese build of Firefox than the actual
> add-ons.
> 
> Hong, can you please look into this?

Chinese extension, either developed by us at Mozilla Online or not, does not appear in recent correlations of this crash, so clear needinfo for Hong.
Flags: needinfo?(htang)
(In reply to Benjamin Smedberg  [:bsmedberg] from comment #17)
> connor, can you contact Avast and see if they have any clues on this one?
> There's not much useful given the data we have other than the correlation.
> It looks like somebody is injecting code and trying to launch a thread but
> failing somehow.

This question is still outstanding.

still an active crash according to https://crash-stats.mozilla.com/report/list?signature=__RtlUserThreadStart%20|%20_RtlUserThreadStart#tab-sigsummary
Flags: needinfo?(mconnor)

Marking this as Resolved > Worksforme since there are no more crashes with this signature in the past 6 months.

Status: NEW → RESOLVED
Closed: 7 months ago
Resolution: --- → WORKSFORME
You need to log in before you can comment on or make changes to this bug.