663466 - crash @ __RtlUserThreadStart | _RtlUserThreadStart

Status: RESOLVED WORKSFORME

(Firefox :: General, defect)

Description

[Marcia Knous [:marcia]]

This bug was filed from the Socorro interface and is 
report bp-40656464-c36a-409f-a8eb-193ac2110608 .
============================================================= 

Seen while reviewing crash stats. https://crash-stats.mozilla.com/report/list?signature=__RtlUserThreadStart to the crashes which span all versions.

Some of the comments mention having trouble with toolbars and a fair number have the addon compatibility extension installed. Addon correlation:

78% (39/50) vs.   1% (364/27164) [e-mail]
52% (26/50) vs.   0% (26/27164) {4D1E692F-D179-413b-A987-EEEAAD85DDB3}
52% (26/50) vs.   0% (26/27164) [e-mail] (Photobucket Uploader, https://addons.mozilla.org/addon/10035)
52% (26/50) vs.   0% (27/27164) [e-mail]
52% (26/50) vs.   0% (28/27164) [e-mail]
52% (26/50) vs.   0% (28/27164) {f92a9fe4-2850-4198-b9d5-279880e49b16}
52% (26/50) vs.   0% (32/27164) [e-mail]
52% (26/50) vs.   0% (42/27164) {8A9386B4-E958-4c4c-ADF4-8F26DB3E4829} (PriceGong, https://addons.mozilla.org/addon/10433)
52% (26/50) vs.   1% (152/27164) {1E73965B-8B48-48be-9C8D-68B920ABC1C4}
52% (26/50) vs.   1% (302/27164) {635abd67-4fe9-1b23-4f01-e679fa7484c1} (Yahoo! Toolbar, https://addons.mozilla.org/addon/2032)
52% (26/50) vs.   1% (389/27164) avg@igeared
52% (26/50) vs.   2% (524/27164) {CAFEEFAC-0016-0000-0026-ABCDEFFEDCBA}
52% (26/50) vs.   2% (641/27164) [e-mail]
36% (18/50) vs.   0% (21/27164) {32a1fd71-835e-4b11-8e54-886fda0b4c89}
36% (18/50) vs.   3% (907/27164) [e-mail] (Personas, https://addons.mozilla.org/addon/10900)
52% (26/50) vs.  22% (6000/27164) {CAFEEFAC-0016-0000-0024-ABCDEFFEDCBA}
26% (13/50) vs.   0% (14/27164) [e-mail]
26% (13/50) vs.   0% (15/27164) [e-mail] (Add-on Collector, https://addons.mozilla.org/addon/11950)
26% (13/50) vs.   0% (20/27164) [e-mail]
26% (13/50) vs.   0% (32/27164) {cd617375-6743-4ee8-bac4-fbf10f35729e} (RightToClick, https://addons.mozilla.org/addon/12572)
26% (13/50) vs.   0% (35/27164) [e-mail] (Russian spellchecking dictionary, https://addons.mozilla.org/addon/3703)
26% (13/50) vs.   0% (41/27164) [e-mail] (Fast Dial, https://addons.mozilla.org/addon/5721)
26% (13/50) vs.   0% (46/27164) [e-mail]
26% (13/50) vs.   0% (63/27164) SkipScreen@SkipScreen (SkipScreen, https://addons.mozilla.org/addon/11243)
26% (13/50) vs.   2% (466/27164) {19503e42-ca3c-4c27-b1e2-9cdb2170ee34} (FlashGot, https://addons.mozilla.org/addon/220)
26% (13/50) vs.   3% (887/27164) {D4DD63FA-01E4-46a7-B6B1-EDAB7D6AD389} (Download Statusbar, https://addons.mozilla.org/addon/26)
32% (16/50) vs.  10% (2777/27164) {d10d0bf8-f5b5-c8b4-a8b2-2b9879e08c5d} (Adblock Plus, https://addons.mozilla.org/addon/1865)
26% (13/50) vs.   6% (1623/27164) {CAFEEFAC-0016-0000-0025-ABCDEFFEDCBA}
12% (6/50) vs.   0% (32/27164) {37964A3C-4EE8-47b1-8321-34DE2C39BA4D}
92% (46/50) vs.  82% (22208/27164) [e-mail] (Mozilla Labs - Test Pilot, https://addons.mozilla.org/addon/13661)
10% (5/50) vs.   0% (5/27164) {d21e1d10-117c-11df-8a39-0800200c9a66}
10% (5/50) vs.   0% (8/27164) {6236BA26-C117-4007-928C-DE0716C7FA78} (Chameleon Bob: layouts for Facebook, https://addons.mozilla.org/addon/11584)
10% (5/50) vs.   0% (9/27164) {6236BA26-C117-4007-928C-DE0716C7FA68} (Express Tab, https://addons.mozilla.org/addon/14584)
10% (5/50) vs.   0% (14/27164) {6236BA26-C117-4007-928C-DE0716C7FA48}
10% (5/50) vs.   0% (24/27164) {6236BA26-C117-4007-928C-DE0716C7FA38}
10% (5/50) vs.   0% (92/27164) {3e9a3920-1b27-11da-8cd6-0800200c9a66}
14% (7/50) vs.   8% (2174/27164) {CAFEEFAC-0016-0000-0023-ABCDEFFEDCBA}

https://crash-stats.mozilla.com/report/index/40656464-c36a-409f-a8eb-193ac2110608

Frame 	Module 	Signature [Expand] 	Source
0 		@0x3f90000 	
1 	ntdll.dll 	__RtlUserThreadStart 	
2 	ntdll.dll 	_RtlUserThreadStart

Comment 1

[Sheila Mooney]

Appears at the #26 spot on 8.0 and #18 on 8.0.1 over the past week.

Comment 2

[Scoobidiver (away)]

It's in the long list of bugs correlated to Спутник @Mail.Ru and Yandex.Bar:
68% (101/148) vs.   7% (3725/51702) [e-mail] (Yandex.Bar, https://addons.mozilla.org/addon/3495)
64% (94/148) vs.   6% (3010/51702) {37964A3C-4EE8-47b1-8321-34DE2C39BA4D}

Comment 3

[Sheila Mooney]

Around 92% of these are occurring < 1 min.

Comment 4

[Sheila Mooney]

This seems to have gone way down in volume. Sitting at #63 on Fx12. Removing the top crash keyword.

Comment 5

[Scoobidiver (away)]

It's #27 browser crasher in 21.0, #14 in 22.0b3 and 23.0a2, and #40 in 24.0a1.

Windows XP is not affected. 93% of crashes are at startup.

Here are interesting correlations:
* 21.0:
  __RtlUserThreadStart | _RtlUserThreadStart|EXCEPTION_ACCESS_VIOLATION_EXEC (270 crashes)
    64% (173/270) vs.  12% (13495/111807) aswJsFlt.dll (Avast! Antivirus Script Blocking filter library)
          3% (8/270) vs.   0% (398/111807) 6.0.1367.0
          0% (1/270) vs.   0% (23/111807) 7.0.1451.402
          1% (4/270) vs.   0% (188/111807) 7.0.1455.186
          1% (3/270) vs.   0% (369/111807) 7.0.1456.418
          2% (6/270) vs.   1% (846/111807) 7.0.1466.549
         15% (40/270) vs.   2% (2717/111807) 7.0.1474.765
         17% (46/270) vs.   3% (3571/111807) 8.0.1483.72
          1% (4/270) vs.   0% (333/111807) 8.0.1488.286
         22% (60/270) vs.   4% (4040/111807) 8.0.1489.300
    66% (177/270) vs.  17% (18596/111807) snxhk.dll (Avast! Antivirus)
          0% (1/270) vs.   0% (125/111807) 6.0.1125.0
          3% (8/270) vs.   0% (415/111807) 6.0.1367.0
          0% (1/270) vs.   0% (162/111807) 6.0.999.0
          0% (1/270) vs.   0% (25/111807) 7.0.1451.402
          1% (4/270) vs.   0% (209/111807) 7.0.1455.186
          1% (3/270) vs.   0% (384/111807) 7.0.1456.418
          2% (6/270) vs.   1% (887/111807) 7.0.1466.549
         15% (40/270) vs.   3% (2851/111807) 7.0.1474.765
          0% (1/270) vs.   0% (339/111807) 8.0.1482.45
         17% (46/270) vs.   5% (5265/111807) 8.0.1483.72
          1% (4/270) vs.   0% (484/111807) 8.0.1488.286
         23% (62/270) vs.   6% (6378/111807) 8.0.1489.300
     52% (141/270) vs.   8% (8536/111807) BrowserProtect.dll (PerformerSoft Protector)
          1% (3/270) vs.   1% (752/111807) 2.6.1095.52
         51% (138/270) vs.   6% (7065/111807) 2.6.1249.132
     24% (66/270) vs.   2% (2608/111807) Iminent.WinCore.dll (Iminent)
          1% (2/270) vs.   0% (178/111807) 3.47.0.0
          4% (12/270) vs.   0% (89/111807) 5.18.52.0
          0% (1/270) vs.   0% (44/111807) 5.26.21.0
          0% (1/270) vs.   0% (64/111807) 5.29.41.0
          0% (1/270) vs.   0% (74/111807) 5.35.51.0
          1% (2/270) vs.   0% (22/111807) 5.48.42.0
          1% (2/270) vs.   0% (47/111807) 5.51.31.0
          0% (1/270) vs.   0% (96/111807) 5.52.31.0
          2% (5/270) vs.   0% (325/111807) 6.14.22.0
          1% (4/270) vs.   0% (234/111807) 6.17.41.0
         12% (32/270) vs.   0% (225/111807) 6.21.22.0
          1% (3/270) vs.   1% (708/111807) 6.4.56.0
     18% (48/270) vs.   4% (4215/111807) mgAdaptersProxy.dll (SweetIM Messenger)
          1% (3/270) vs.   0% (99/111807) 3.3.0.7
          0% (1/270) vs.   0% (58/111807) 3.4.0.5
          1% (2/270) vs.   0% (35/111807) 3.5.0.8
          1% (4/270) vs.   0% (301/111807) 3.6.0.2
          5% (13/270) vs.   0% (296/111807) 3.6.0.8
          2% (6/270) vs.   1% (771/111807) 3.7.0.5
          7% (19/270) vs.   2% (2304/111807) 3.7.0.7

* 22.0 Beta:
  __RtlUserThreadStart | _RtlUserThreadStart|EXCEPTION_ACCESS_VIOLATION_WRITE (323 crashes)
     33% (106/323) vs.   1% (515/37606) datamngr.dll (Bandoo's Data Manager)
         33% (106/323) vs.   1% (478/37606) 1.0.0.1
     14% (44/323) vs.   1% (326/37606) McSACorePS.dll (McAfee SiteAdvisor)
         14% (44/323) vs.   1% (260/37606) 3.6.0.137
          0% (0/323) vs.   0% (3/37606) 3.6.1.106
     14% (44/323) vs.   1% (329/37606) saplugin.dll (McAfee SiteAdvisor)
         14% (44/323) vs.   1% (267/37606) 3.6.0.187
     14% (44/323) vs.   2% (759/37606) sahook.dll (McAfee SiteAdvisor)
         14% (44/323) vs.   1% (489/37606) 3.6.0.187
          0% (0/323) vs.   0% (155/37606) 3.6.1.193

* 23.0a2:
  __RtlUserThreadStart | _RtlUserThreadStart|EXCEPTION_ACCESS_VIOLATION_WRITE (25 crashes)
     44% (11/25) vs.   2% (37/2095) avsda.dll (Avira Free Antivirus)
         44% (11/25) vs.   1% (28/2095) 13.6.2.666
     24% (6/25) vs.   0% (6/2095) KainyX.dll

I don't experience a startup crash with Avast! Antivirus Basic Edition with or without the extension in 21.0.

More reports at:
https://crash-stats.mozilla.com/report/list?signature=__RtlUserThreadStart+|+_RtlUserThreadStart

Comment 6

[Alex Keybl [:akeybl]]

We have a strong suspicion that this is a 3rd party issue since this wasn't a top crash while FF22 was on Beta.

One thing we saw in the crash comments that really concerned us was:

"any ideas, each time i open it i see the open in safe mode prompt for a split second then it goes straight to crash report and restarting firefox or quiting firefox doesnt seem to do anything"

Comment 7

[Alex Keybl [:akeybl]]

That needinfo was meant to ask whether Kairo could check how many unique users are affected on each channel.

Comment 8

[Alex Keybl [:akeybl]]

Including MattN, since he worked on bug 294260, and may be able to shine some light onto what situations would warrant the crash comment from comment 6. Especially since we think this was tickled by a third party.

Comment 9

[Matthew N. [:MattN]]

(In reply to Alex Keybl [:akeybl] from comment #6)
> One thing we saw in the crash comments that really concerned us was:
> 
> "any ideas, each time i open it i see the open in safe mode prompt for a
> split second then it goes straight to crash report and restarting firefox or
> quiting firefox doesnt seem to do anything"

I suspect it's just the startup crash detection doing its job but we are still hitting the crash in safe mode while the dialog is open. This elimates plugins as the cause because they shouldn't be started that early.

Comment 10

[Robert Kaiser]

(In reply to Alex Keybl [:akeybl] from comment #7)
> That needinfo was meant to ask whether Kairo could check how many unique
> users are affected on each channel.

breakpad=> SELECT version,COUNT(*) as crashes,COUNT(DISTINCT client_crash_date - install_age  * interval '1 second') as installations FROM reports WHERE product='Firefox' AND signature='__RtlUserThreadStart | _RtlUserThreadStart' AND date_processed BETWEEN '2013-05-29' AND '2013-06-05' GROUP BY version;
  version  | crashes | installations 
-----------+---------+---------------
[...]
 21.0      |    2796 |          2014
 22.0      |    1184 |           262
 23.0a2    |      79 |            21
 24.0a1    |      21 |             7
[...]

Starting next week, we should get those numbers in Signature Summaries on Socorro as well.

Comment 11

[Alex Keybl [:akeybl]]

Gavin - any suggestion on who could look at this startup crash? I believe Matt's comment 9 suggests this is an issue that's likely completely on our side.

Comment 12

[:Gavin Sharp [email: [e-mail]]]

(In reply to Alex Keybl [:akeybl] from comment #11)
> I believe Matt's comment 9 suggests this is an issue that's likely completely on our
> side.

He said "not plugins", which doesn't necessarily imply "completely on our side" :)

I don't have any ideas here, I think bsmedberg is a better wild-guess owner than I am.

Comment 13

[Benjamin Smedberg]

What's the current rank of this? It's unlikely that I'll be able to work on this for a couple weeks at least.

Most of these appear to be Flash-process crashes that are not labeled as such. See e.g. https://crash-stats.mozilla.com/report/index/53347fea-bcf4-4e10-b87f-aa1062130614

It would be helpful if somebody could classify these by process by searching the module list for firefox.exe or plugin-container.exe or FlashPlayerPlugin*.exe

Comment 14

[Scoobidiver (away)]

A comment in French says that Firefox crashes (at startup) after a Skype update.
Most comments are in Portuguese.

(In reply to Benjamin Smedberg  [:bsmedberg] from comment #13)
> What's the current rank of this?
It's #15 top browser crasher in 21.0, #17 in 22.0b5 and 23.0a2. It jumps six ranks in one week.
For the last day, it's even #10 top crasher in 21.0.

> Most of these appear to be Flash-process crashes that are not labeled as
> such. See e.g.
> https://crash-stats.mozilla.com/report/index/53347fea-bcf4-4e10-b87f-
> aa1062130614
This one is bug 852096 and doesn't have the same correlations as this one. Compare comment 5 and bug 852096.

> It would be helpful if somebody could classify these by process by searching
> the module list for firefox.exe or plugin-container.exe or
> FlashPlayerPlugin*.exe
Here they are for 21.0:
  __RtlUserThreadStart | _RtlUserThreadStart|EXCEPTION_ACCESS_VIOLATION_EXEC (602 crashes)
    100% (601/602) vs.  69% (69637/101515) firefox.exe
No plugin-container.exe or FlashPlayerPlugin_*.exe correlations.

Comment 15

[Scoobidiver (away)]

(In reply to Scoobidiver from comment #14)
> A comment in French says that Firefox crashes (at startup) after a Skype
> update.
Skype 6.5 was released on June 5 (second spike) and 6.3 on May 13 (first spike in comment 5). See http://blogs.skype.com/category/garage-updates/

Comment 16

[Scoobidiver (away)]

It's #4 top browser crasher in the first days of 22.0 and still correlated to Avast Antivirus.

Comment 17

[Benjamin Smedberg]

connor, can you contact Avast and see if they have any clues on this one? There's not much useful given the data we have other than the correlation. It looks like somebody is injecting code and trying to launch a thread but failing somehow.

Comment 18

[Lukas Blakk [:lsblakk] use ?needinfo]

#15 on 23.0b1 and going down so untracking.

Comment 19

[John Hesling [:John99] (NeedInfo me)]

#8 in Fx 23.0 Release

Comment 20

[Robert Kaiser]

Current correlations:

Addons:
  __RtlUserThreadStart | _RtlUserThreadStart|EXCEPTION_ACCESS_VIOLATION_EXEC (425 crashes)
     66% (280/425) vs.   1% (1257/95359) {1B33E42F-EF14-4cd3-B6DC-174571C4349C}
     43% (181/425) vs.   1% (1298/95359) [e-mail]
     42% (179/425) vs.   1% (1320/95359) [e-mail]
     41% (175/425) vs.   1% (1185/95359) [e-mail]
     39% (165/425) vs.   1% (1264/95359) [e-mail]
     39% (164/425) vs.   1% (1228/95359) [e-mail]
     37% (157/425) vs.   1% (1044/95359) [e-mail]
     36% (155/425) vs.   1% (1103/95359) [e-mail]
     35% (150/425) vs.   1% (912/95359) [e-mail]
     33% (142/425) vs.   1% (843/95359) [e-mail]
     33% (140/425) vs.   1% (855/95359) [e-mail]
     31% (132/425) vs.   1% (1024/95359) [e-mail]
     14% (60/425) vs.   0% (321/95359) [e-mail]
     12% (49/425) vs.   1% (567/95359) [e-mail]
      9% (38/425) vs.   0% (185/95359) {3ec6f6a6-94e7-41d5-8c36-cec334456e1e}
      7% (30/425) vs.   0% (242/95359) [e-mail]
      7% (30/425) vs.   1% (947/95359) [e-mail] (Personas, https://addons.mozilla.org/addon/10900)
      6% (26/425) vs.   0% (218/95359) [e-mail]
      6% (25/425) vs.   0% (334/95359) [e-mail] (NetVideoHunter Video Downloader, https://addons.mozilla.org/addon/7447)

Modules:
  __RtlUserThreadStart | _RtlUserThreadStart|EXCEPTION_ACCESS_VIOLATION_EXEC (425 crashes)
     99% (419/425) vs.  42% (39881/95359) DWrite.dll
     96% (410/425) vs.  40% (38120/95359) cryptsp.dll
     87% (370/425) vs.  39% (37657/95359) nlaapi.dll
     87% (370/425) vs.  40% (37672/95359) NapiNSP.dll
     99% (422/425) vs.  52% (49808/95359) ntmarta.dll
     82% (348/425) vs.  35% (33544/95359) explorerframe.dll
     72% (305/425) vs.  25% (23918/95359) d2d1.dll
     82% (348/425) vs.  35% (33568/95359) dui70.dll
     86% (366/425) vs.  39% (37618/95359) pnrpnsp.dll
     73% (311/425) vs.  27% (25303/95359) dxgi.dll
     83% (354/425) vs.  37% (35254/95359) FWPUCLNT.DLL
     82% (348/425) vs.  36% (34383/95359) duser.dll
     66% (280/425) vs.  25% (23899/95359) d3d10_1core.dll
     66% (280/425) vs.  25% (23899/95359) d3d10_1.dll
     98% (415/425) vs.  58% (55059/95359) samcli.dll
     98% (415/425) vs.  58% (55082/95359) wkscli.dll
     98% (415/425) vs.  58% (55085/95359) netutils.dll
     98% (415/425) vs.  58% (55112/95359) srvcli.dll
    100% (425/425) vs.  62% (58722/95359) propsys.dll
     96% (410/425) vs.  59% (56296/95359) profapi.dll
     38% (163/425) vs.   1% (1085/95359) oledlg.dll
     58% (246/425) vs.  21% (19746/95359) winhttp.dll
     38% (162/425) vs.   1% (930/95359) MinHook.32.dll
     91% (385/425) vs.  54% (51065/95359) sspicli.dll
     37% (159/425) vs.   1% (579/95359) msimtf.dll
     98% (415/425) vs.  61% (58234/95359) devobj.dll
     37% (159/425) vs.   1% (892/95359) npcoba32.dll
     98% (415/425) vs.  61% (58352/95359) KERNELBASE.dll
     98% (415/425) vs.  61% (58353/95359) sechost.dll
     98% (415/425) vs.  61% (58353/95359) CRYPTBASE.dll
     53% (224/425) vs.  17% (15762/95359) oleacc.dll
     39% (166/425) vs.   3% (3194/95359) mshtml.dll
     98% (415/425) vs.  62% (59085/95359) cfgmgr32.dll
    100% (425/425) vs.  65% (61825/95359) winnsi.dll
    100% (425/425) vs.  65% (61843/95359) IPHLPAPI.DLL


Looking at the add-ons, this looks very heavily correlated with the Mozilla China (mozillaonline.com) add-ons. It may be caused by something they ship there or by some other Chinese thing. We need input from them. Release Management, do you know who to ask there?

Comment 21

[Lukas Blakk [:lsblakk] use ?needinfo]

Let's ask Jorge first since he's more familiar with addons community

Comment 22

[Jorge Villalobos [:jorgev] (he/him)]

{1B33E42F-EF14-4cd3-B6DC-174571C4349C} and [e-mail] are also Chinese add-ons, but not developed by Mozilla Online, so it's possible that this crash has more to do with the Chinese build of Firefox than the actual add-ons.

Hong, can you please look into this?

Comment 23

[Mark Hammond [:markh] [:mhammond]]

The crash here looks a little similar to bug 902790, and the symptoms might also match (ie, that Avast or something is preventing the plugin child process from starting)?

Comment 24

[Robert Kaiser]

The "or something" is the big question.

Comment 25

[Tracy Walker [:tracy]]

this signature drifts in and out of topcrash status

topcrash is being replaced by more precise keywords per https://bugzilla.mozilla.org/show_bug.cgi?id=927557#c3

Comment 26

[Hector Zhao [:hectorz]]

(In reply to Jorge Villalobos [:jorgev] from comment #22)
> {1B33E42F-EF14-4cd3-B6DC-174571C4349C} and [e-mail] are also
> Chinese add-ons, but not developed by Mozilla Online, so it's possible that
> this crash has more to do with the Chinese build of Firefox than the actual
> add-ons.
> 
> Hong, can you please look into this?

Chinese extension, either developed by us at Mozilla Online or not, does not appear in recent correlations of this crash, so clear needinfo for Hong.

Comment 27

[Wayne Mery (:wsmwk)]

(In reply to Benjamin Smedberg  [:bsmedberg] from comment #17)
> connor, can you contact Avast and see if they have any clues on this one?
> There's not much useful given the data we have other than the correlation.
> It looks like somebody is injecting code and trying to launch a thread but
> failing somehow.

This question is still outstanding.

still an active crash according to https://crash-stats.mozilla.com/report/list?signature=__RtlUserThreadStart%20|%20_RtlUserThreadStart#tab-sigsummary

Comment 28

[Andrei Purice]

Marking this as Resolved > Worksforme since there are no more crashes with this signature in the past 6 months.