535873 - viewpoint plugin incompatible and causes Firefox Crash [@ GetRealWindowOwner ]

Status: RESOLVED FIXED

(Plugins Graveyard :: Viewpoint Media Player, defect)

Description

[chris hofmann]

early 3.6b5 data shows #8 topcrash as

https://crash-stats.mozilla.com/report/index/918e24d4-7bac-4187-9763-568422091218

Frame  	Module  	Signature [Expand]  	Source
0 		@0x10cd64 	
1 	user32.dll 	GetRealWindowOwner 	
2 	user32.dll 	DispatchMessageWorker 	
3 	user32.dll 	DispatchMessageW 	
4 	xul.dll 	nsBaseAppShell::OnProcessNextEvent 	widget/src/xpwidgets/nsBaseAppShell.cpp:278
5 	xul.dll 	nsTArray_base::ShiftData 	obj-firefox/xpcom/build/nsTArray.cpp:162
6 	xul.dll 	nsBaseAppShell::Run 	widget/src/xpwidgets/nsBaseAppShell.cpp:170
7 	xul.dll 	nsAppStartup::Run 	toolkit/components/startup/src/nsAppStartup.cpp:182
8 	nspr4.dll 	PR_GetEnv 	
9 	firefox.exe 	wmain 	toolkit/xre/nsWindowsWMain.cpp:120
10 	firefox.exe 	__tmainCRTStartup 	obj-firefox/memory/jemalloc/crtsrc/crtexe.c:591
11 	kernel32.dll 	BaseThreadInitThunk 	
12 	ntdll.dll 	__RtlUserThreadStart 	
13 	ntdll.dll 	_RtlUserThreadStart

more reports at

https://crash-stats.mozilla.com/report/list?version=Firefox%3A3.6b5&query_search=signature&query_type=exact&query=&date=&range_value=2&range_unit=weeks&do_query=&signature=GetRealWindowOwner&page=1

only source changes on the stack that have changed recently are pretty far from the top of the stack but are shown as

1d7c21b6ca52
2009-11-03 22:35 -0800	Vladimir Vukicevic - b=524904, add support for generic Windows DLL blocklist; r=shaver

and I'm guessing are unrelated, unless the have introduce some timing problem

The signature is is also in beta4 but currently ranks #173 and we seem to see this signature in spikes, maybe around each release.

56  crashes for GetRealWindowOwner on  20091101-crashdata
43  crashes for GetRealWindowOwner on  20091102-crashdata
50  crashes for GetRealWindowOwner on  20091103-crashdata
50  crashes for GetRealWindowOwner on  20091104-crashdata
56  crashes for GetRealWindowOwner on  20091105-crashdata
96  crashes for GetRealWindowOwner on  20091106-crashdata
50  crashes for GetRealWindowOwner on  20091107-crashdata
70  crashes for GetRealWindowOwner on  20091108-crashdata
64  crashes for GetRealWindowOwner on  20091109-crashdata
75  crashes for GetRealWindowOwner on  20091110-crashdata
77  crashes for GetRealWindowOwner on  20091111-crashdata
64  crashes for GetRealWindowOwner on  20091112-crashdata
65  crashes for GetRealWindowOwner on  20091113-crashdata
78  crashes for GetRealWindowOwner on  20091114-crashdata
74  crashes for GetRealWindowOwner on  20091115-crashdata
76  crashes for GetRealWindowOwner on  20091116-crashdata
70  crashes for GetRealWindowOwner on  20091117-crashdata
95  crashes for GetRealWindowOwner on  20091118-crashdata
65  crashes for GetRealWindowOwner on  20091119-crashdata
68  crashes for GetRealWindowOwner on  20091120-crashdata
92  crashes for GetRealWindowOwner on  20091121-crashdata
84  crashes for GetRealWindowOwner on  20091122-crashdata
89  crashes for GetRealWindowOwner on  20091123-crashdata
79  crashes for GetRealWindowOwner on  20091124-crashdata
76  crashes for GetRealWindowOwner on  20091125-crashdata
68  crashes for GetRealWindowOwner on  20091126-crashdata
95  crashes for GetRealWindowOwner on  20091127-crashdata
80  crashes for GetRealWindowOwner on  20091128-crashdata
80  crashes for GetRealWindowOwner on  20091129-crashdata
92  crashes for GetRealWindowOwner on  20091130-crashdata
78  crashes for GetRealWindowOwner on  20091201-crashdata
71  crashes for GetRealWindowOwner on  20091202-crashdata
78  crashes for GetRealWindowOwner on  20091203-crashdata
91  crashes for GetRealWindowOwner on  20091204-crashdata
115  crashes for GetRealWindowOwner on  20091205-crashdata
79  crashes for GetRealWindowOwner on  20091206-crashdata
70  crashes for GetRealWindowOwner on  20091207-crashdata
111  crashes for GetRealWindowOwner on  20091208-crashdata
69  crashes for GetRealWindowOwner on  20091209-crashdata
99  crashes for GetRealWindowOwner on  20091210-crashdata
89  crashes for GetRealWindowOwner on  20091211-crashdata
80  crashes for GetRealWindowOwner on  20091212-crashdata
94  crashes for GetRealWindowOwner on  20091213-crashdata
87  crashes for GetRealWindowOwner on  20091214-crashdata
108  crashes for GetRealWindowOwner on  20091215-crashdata
106  crashes for GetRealWindowOwner on  20091216-crashdata
94  crashes for GetRealWindowOwner on  20091217-crashdata

checking --- 20091217-crashdata.csv GetRealWindowOwner
release total-crashes
              GetRealWindowOwner crashes
                         pct.
all     227066  94      0.000413977
3.0.15  17135           0
3.0.16  22875           0
3.5.5   47876   17      0.000355084
3.5.6   80444   40      0.00049724
3.6b5   990     9       0.00909091
3.6b4   21208   23      0.0010845
3.6b3   670             0
3.6b2   739             0
3.6b1   2144            0

Comment 1

[chris hofmann]

the crashes are mostly close to start up/restart with a few outliers, and very concentrated on NT6.1.7600  

os breakdown
90      0.957447        Windows NT6.1.7600
4       0.0425532       Windows NT6.1.7100

domains of sites
  35 chrome://ietab
  33 //
   6 https://components.viewpoint.com
   4 http://ting.blog.gougou.com
   2 chrome://coralietab
   2 \N//

Comment 2

[chris hofmann]

there are a pretty wide set of addons in play even on the small set of crashes in recent 3.6b4 data from 2009 12 15

  GetRealWindowOwner|EXCEPTION_ACCESS_VIOLATION (22 crashes)
     45% (10/22) vs.   0% (63/20478) {1B33E42F-EF14-4cd3-B6DC-174571C4349C}
     45% (10/22) vs.   1% (194/20478) [e-mail] (Coral IE Tab, https://addons.mozilla.org/addon/10909)
     45% (10/22) vs.   3% (514/20478) {19503e42-ca3c-4c27-b1e2-9cdb2170ee34} (FlashGot, https://addons.mozilla.org/addon/220)
     23% (5/22) vs.   0% (6/20478) {0851d9cd-87db-4a0d-a792-097dc9071486} (DownloadStudio Integration, https://addons.mozilla.org/addon/627)
     23% (5/22) vs.   1% (174/20478) [e-mail] (1-Click YouTube Video Downloader, https://addons.mozilla.org/addon/13990)
     27% (6/22) vs.   5% (1116/20478) {3112ca9c-de6d-4884-a869-9855de68056c} (Google Toolbar, https://addons.mozilla.org/addon/6249)
     23% (5/22) vs.   1% (283/20478) [e-mail]
     32% (7/22) vs.  11% (2187/20478) {d10d0bf8-f5b5-c8b4-a8b2-2b9879e08c5d} (Adblock Plus, https://addons.mozilla.org/addon/1865)
     18% (4/22) vs.   4% (908/20478) {D4DD63FA-01E4-46a7-B6B1-EDAB7D6AD389} (Download Statusbar, https://addons.mozilla.org/addon/26)
     14% (3/22) vs.   0% (14/20478) {b3f91530-1905-11de-8c30-0800200c9a66} (Vfox3_Basic, https://addons.mozilla.org/addon/11227)
     14% (3/22) vs.   0% (26/20478) [e-mail]
     14% (3/22) vs.   0% (44/20478) {113c2360-15a3-11de-8c30-0800200c9a66} (Vfox3, https://addons.mozilla.org/addon/11164)
     14% (3/22) vs.   0% (65/20478) {00352F14-3F76-4e4d-ACFF-9976D7E4B3B9} (MacOSX Theme, https://addons.mozilla.org/addon/7172)
     14% (3/22) vs.   1% (165/20478) {64161300-e22b-11db-8314-0800200c9a66} (Speed Dial, https://addons.mozilla.org/addon/4810)
     14% (3/22) vs.   2% (339/20478) [e-mail] (FastestFox - Browse Faster, https://addons.mozilla.org/addon/9825)
     14% (3/22) vs.   3% (642/20478) {1018e4d6-728f-4b20-ad56-37578a4de76b} (Flagfox, https://addons.mozilla.org/addon/5791)
      9% (2/22) vs.   0% (3/20478) [e-mail] (DblClicker, https://addons.mozilla.org/addon/13902)
      9% (2/22) vs.   0% (6/20478) [e-mail] (Informational Tab, https://addons.mozilla.org/addon/4930)
      9% (2/22) vs.   0% (6/20478) {cb84136f-9c44-433a-9048-c5cd9df1dc16} (Browser Defender Toolbar, https://addons.mozilla.org/addon/8909)
      9% (2/22) vs.   0% (10/20478) {e9259cba-e7ad-4f74-863f-ef9fe935394d}
      9% (2/22) vs.   0% (11/20478) [e-mail] (Open Bookmarks in New Tab, https://addons.mozilla.org/addon/13784)
      9% (2/22) vs.   0% (21/20478) [e-mail] (Tabberwocky, https://addons.mozilla.org/addon/14439)
      9% (2/22) vs.   1% (103/20478) [e-mail] (Strata40, https://addons.mozilla.org/addon/14284)
      9% (2/22) vs.   1% (123/20478) [e-mail] (Personal Menu, https://addons.mozilla.org/addon/3895)
      9% (2/22) vs.   1% (173/20478) [e-mail] (FireGestures, https://addons.mozilla.org/addon/6366)
      9% (2/22) vs.   1% (174/20478) {1280606b-2510-4fe0-97ef-9b5a22eafe41} (Fission, https://addons.mozilla.org/addon/1951)
      9% (2/22) vs.   1% (175/20478) [e-mail] (Adblock Plus: Element Hiding Helper, https://addons.mozilla.org/addon/4364)
     23% (5/22) vs.  15% (3106/20478) {CAFEEFAC-0016-0000-0017-ABCDEFFEDCBA}
      9% (2/22) vs.   2% (378/20478) {dc572301-7619-498c-a57d-39143191b318} (Tab Mix Plus, https://addons.mozilla.org/addon/1122)
      9% (2/22) vs.   2% (379/20478) {0329E7D6-6F54-462D-93F6-F5C3118BADF2}
      9% (2/22) vs.   2% (411/20478) {a0d7ccb3-214d-498b-b4aa-0e8fda9a7bf7} (WOT, https://addons.mozilla.org/addon/3456)
      9% (2/22) vs.   3% (557/20478) {F17C1572-C9EC-4e5c-A542-D05CBB5C5A08} (Download Accelerator Plus Component, https://addons.mozilla.org/addon/11720)
      9% (2/22) vs.   3% (636/20478) {CAFEEFAC-0016-0000-0016-ABCDEFFEDCBA} (Java Console, http://java.sun.com/javase/downloads/)

maybe some kind of contention for first run page slotting when many addons get triggered for updates?   that would be a case that happens for many users around update restarts.

Comment 3

[Marcia Knous [:marcia]]

I crashed during the FFT run on Windows 7 in this stack and can reproduce 100%.

STR:
1. http://www.viewpoint.com/technologies/viewpoint-media-player.shtml
2. Click the download link. When I click the file name in the download manager, I crash.

http://crash-stats.mozilla.com/report/index/bp-291dd72c-fa33-4b8d-bd1a-839d92100111

Comment 4

[chris hofmann]

more prominent in 3.6 and seems to be seen mostly/only Windows NT6.1

checking --- 20100110-crashdata.csv GetRealWindowOwner
release total-crashes
              GetRealWindowOwner crashes
                         pct.
all     215352  120     0.000557227
3.0.15  1958    2       0.00102145
3.0.16  4940            0
3.5.5   5270    3       0.00056926
3.5.6   16510   7       0.000423985
3.5.7   102593  58      0.000565341
3.6     10425   37      0.00354916
3.6b5   14884   9       0.000604676
3.6b4   1505            0
3.6b3   721     2       0.00277393
3.6b2   729             0
3.6b1   2102    2       0.000951475


os breakdown
113     0.941667        Windows NT6.1.7600
7       0.0583333       Windows NT6.1.7100

Comment 5

[chris hofmann]

definitely wanted and maybe should block since it seems to be higher frequency on 3.6 than should be expected.

Comment 6

[chris hofmann]

many users also see this crash with IE tab installed and think it responsible. so maybe its also tickling similar code as marcia found in comment 3 STR.

domains of sites
  36 chrome://ietab
  27 //
   8 file://
   5 \N//
   4 https://components.viewpoint.com
   3 http://www.youtube.com
   3 http://gallery.popularscreensavers.com
   2 http://www3.interscience.wiley.com
   2 http://www1.suzuki.co.jp
   2 http://www.insulation.kingspan.com
   2 http://www.ingyentv.hu
   2 http://www.astonmartin.com
   2 chrome://coralietab
   2 about:blank//

Comment 7

[Mike Beltzner [:beltzner, not reading bugmail]]

By "maybe should block" you mean "is a high enough frequency to require a respin and delay for delivery?" Based on the data in comment 4 I think that this is a good candidate for investigation as part of crashkill, but doesn't feel like a release blocker to me.

(for those who don't have a translation map handy, NT6.1 = Windows 7 or Windows Server 2008)

Comment 8

[chris hofmann]

By "maybe should block" I mean that 
now that it has STR, 
and if we get can get a quick fix, 
it should be considered for a "ride-along"

Comment 9

[Mike Beltzner [:beltzner, not reading bugmail]]

OK, I agree with comment 8 entirely. Marking [3.6.x] to pick up those flags when we create them a little later this week. When we have a fix on there, feel free to add [rc-ridealong] to the whiteboard.

Comment 10

[Robert Strong (they/them - no direct email)]

(In reply to comment #3)
> I crashed during the FFT run on Windows 7 in this stack and can reproduce 100%.
> 
> STR:
> 1. http://www.viewpoint.com/technologies/viewpoint-media-player.shtml
> 2. Click the download link. When I click the file name in the download manager,
> I crash.
> 
> http://crash-stats.mozilla.com/report/index/bp-291dd72c-fa33-4b8d-bd1a-839d92100111
I've reproduced this as well and this specific crash appears to be due to the way viewpoint compiled their plugin. Per bug 501429 comment #89 I've tracked this down to the landing of bug 418613. Viewpoint specifically states on their site that the plugin only works with Firefox 2 and I've filed bug 539282 to blocklist it and remove it from pfs.

Comment 11

[Johnny Stenback (:jst)]

Excellent detective work Robert! And yes, I agree, we should flat out block the viepoint media plugin.

Comment 12

[chris hofmann]

there are a ton of variations on the user32.dll@ signature.

this one for example gets 800+ crashes per week.

http://crash-stats.mozilla.com/report/index/64cdbf59-f6cf-4358-87c9-548462100111

Viewpoint Media Player Plug In is required to view the page properly. Installed from the link on the site & restarted Firefox. The page does not load, but instead crashes

Frame  	Module  	Signature [Expand]  	Source
0 		@0x4aef15 	
1 	user32.dll 	user32.dll@0x168e9 	
2 	user32.dll 	user32.dll@0x17d30 	
3 	user32.dll 	user32.dll@0x17df9 	
4 	xul.dll 	nsAppShell::ProcessNextNativeEvent 	widget/src/windows/nsAppShell.cpp:165
5 	winmm.dll 	timeGetTime 	
6 	xul.dll 	nsBaseAppShell::OnProcessNextEvent 	widget/src/xpwidgets/nsBaseAppShell.cpp:296
....

looking at many of the signatures a lot of them follow the similar pattern of crashing somewhere in user32.dll sometime after

  xul.dll 	nsAppShell::ProcessNextNativeEvent 	widget/src/windows/nsAppShell.cpp:165

looks like if we block Viewpoint and google talk plugin v1.0.2.0 over in bug 501429, that will be a big win, but I'm wondering if there is a way to dig out a big list of plugins that are hitting this by examining plugins loaded where we crash anywhere in user32.dll after the call to nsAppShell::ProcessNextNativeEvent


here is the list of 100 user32.dll related signatures

http://crash-stats.mozilla.com/query/query?version=ALL%3AALL&date=&range_value=1&range_unit=weeks&query_search=signature&query_type=contains&query=user32.dll&build_id=&do_query=1

if there is any solution that would make these old plugin work again, or at least not crash, that would be a big win as well.  the user that want to get at the viewpoint plugin content above is going to be disappointed until we can get viewpoint to release a new version.  chances on that might be low if they aren't doing much development work these days.

Comment 13

[chris hofmann]

Attachment: signatures where viewpoint is mentioned in comments or crash url

Comment 14

[Robert Strong (they/them - no direct email)]

In the case of the viewpoint plugin I was unable to determine exactly what it was doing that caused the crash though the bug that made it crash only changes compile flags so I suspect they compiled against us in a manner that is explicitly not supported (e.g. internal / unfrozen api's, etc.). I can say that it consistently crashes after ProcessNextNativeEvent receives the WM_DWMNCRENDERINGCHANGED. I changed the code so ProcessNextNativeEvent bypasses the DWMNCRENDERINGCHANGED and then it crashed shortly after when ProcessNextNativeEvent received WM_TIMER.

I suspect that other plugins that crash in ProcessNextNativeEvent are crashing for at the very least slightly if not entirely different reasons than the viewpoint plugin where a fix to workaround the viewpoint plugin crash would not fix the other plugins. Also, the viewpoint plugin doesn't paint properly on the page used to test the crash from bug 501429 in the 3.0 builds likely due to other changes we've made. The viewpoint plugin is likely failing for different reasons in the same code that dispatches messages to all plugins. In other words, this is a common point where messages are sent to plugins so if a plugin is going to crash due to a message it receives this is where we would see it. I also haven't seen any bug reports regarding the viewpoint plugin not working and it hasn't been working since Firefox 3.0 so I have doubts about whether it is really used all that much at this point though that is just a guess.

Comment 15

[Robert Strong (they/them - no direct email)]

From bug 501429

2008-02-22 03:50 no crash
2008-02-22 04:00 crash

Ted, this appears to be due to your patch from bug 418613

cc'ing Ted in case he has any ideas about this

Comment 16

[(not currently active) Ted Mielczarek]

See bug 501429 comment 92. I'm fairly confident this plugin is tripping DEP.

Comment 18

[Marcia Knous [:marcia]]

https://bugzilla.mozilla.org/show_bug.cgi?id=531551#c20 - on Windows Vista and Windows XP I get a crash in [@ UserCallWinProcCheckWow ], which is different from the stack I was getting on Windows 7. And for some reason on Windows Vista the plugin is not being blocklisted even though Bug 539282 has been fixed.

Comment 19

[Robert Strong (they/them - no direct email)]

You probably haven't downloaded the updated blocklist file to your profile yet.

Comment 20

[Joe Drew (not getting mail)]

Carrying over blocking1.9.3? from bug 535873.

Comment 21

[Johnny Stenback (:jst)]

Per bug 539282 the viewpoint plugin is now blocked. Should we not just mark this bug fixed now?