579558 - Crash in mozilla::FrameLayerBuilder::DrawPaintedLayer

Status: RESOLVED WORKSFORME

(Core :: Web Painting, defect)

Description

[Marcia Knous [:marcia]]

Seen while running  Mozilla/5.0 (Windows; Windows NT 5.1; en-US; rv:2.0b2pre) Gecko/20100716 Minefield/4.0b2pre

STR:
1. Load site in URL.
2. Crash 100% http://crash-stats.mozilla.com/report/index/bp-ff0eff34-76c7-4441-95e9-6b7ab2100716/

Link to 51 other crashes in crash-stats: http://tinyurl.com/2fdbfoz

Frame  	Module  	Signature [Expand]  	Source
0 	xul.dll 	mozilla::FrameLayerBuilder::DrawThebesLayer 	layout/base/FrameLayerBuilder.cpp:1361
1 	xul.dll 	mozilla::layers::BasicThebesLayer::Paint 	gfx/layers/basic/BasicLayers.cpp:352
2 	xul.dll 	mozilla::layers::BasicLayerManager::PaintLayer 	gfx/layers/basic/BasicLayers.cpp:920
3 	xul.dll 	mozilla::layers::BasicLayerManager::PaintLayer 	gfx/layers/basic/BasicLayers.cpp:928
4 	xul.dll 	mozilla::layers::BasicLayerManager::EndTransaction 	gfx/layers/basic/BasicLayers.cpp:833
5 	xul.dll 	nsDisplayList::PaintForFrame 	layout/base/nsDisplayList.cpp:404
6 	xul.dll 	nsLayoutUtils::PaintFrame 	layout/base/nsLayoutUtils.cpp:1337
7 	xul.dll 	PresShell::Paint 	layout/base/nsPresShell.cpp:5891
8 	xul.dll 	nsViewManager::Refresh 	view/src/nsViewManager.cpp:415
9 	xul.dll 	nsViewManager::DispatchEvent 	view/src/nsViewManager.cpp:825
10 	xul.dll 	HandleEvent 	view/src/nsView.cpp:160
11 	xul.dll 	nsWindow::DispatchEvent 	widget/src/windows/nsWindow.cpp:3552
12 	xul.dll 	nsWindow::DispatchWindowEvent 	widget/src/windows/nsWindow.cpp:3580
13 	xul.dll 	nsWindow::OnPaint 	widget/src/windows/nsWindowGfx.cpp:563
14 	xul.dll 	nsWindow::ProcessMessage 	widget/src/windows/nsWindow.cpp:4665
15 	xul.dll 	nsWindow::WindowProc 	widget/src/windows/nsWindow.cpp:4310
16 	user32.dll 	InternalCallWinProc 	
17 	user32.dll 	UserCallWinProcCheckWow 	
18 	user32.dll 	DispatchClientMessage 	
19 	user32.dll 	__fnDWORD 	
20 	ntdll.dll 	KiUserCallbackDispatcher 	
21 	xul.dll 	nsXMLContentSink::HandleEndElement 	content/xml/document/src/nsXMLContentSink.cpp:1155
22 	user32.dll 	DispatchMessageW 	
23 	xul.dll 	nsBaseAppShell::OnProcessNextEvent 	widget/src/xpwidgets/nsBaseAppShell.cpp:294
24 	xul.dll 	MessageLoop::DoWork 	ipc/chromium/src/base/message_loop.cc:433
25 	xul.dll 	nsThread::ProcessNextEvent 	xpcom/threads/nsThread.cpp:517
26 	xul.dll 	mozilla::ipc::MessagePump::Run 	ipc/glue/MessagePump.cpp:118
27 	xul.dll 	xul.dll@0xa5db0b 	
28 	xul.dll 	MessageLoop::RunInternal 	ipc/chromium/src/base/message_loop.cc:219
29 	xul.dll 	MessageLoop::RunHandler 	ipc/chromium/src/base/message_loop.cc:202
30 	xul.dll 	xul.dll@0x322bd3 	
31 	xul.dll 	MessageLoop::Run 	ipc/chromium/src/base/message_loop.cc:176
32 	xul.dll 	nsBaseAppShell::Run 	widget/src/xpwidgets/nsBaseAppShell.cpp:175

Comment 1

[Marcia Knous [:marcia]]

This site seems to load fine using the previous day's nightly, Mozilla/5.0 (Windows; Windows NT 5.1; en-US; rv:2.0b2pre) Gecko/20100715 Minefield/4.0b2pre.

Mozilla/5.0 (Windows; Windows NT 5.1; en-US; rv:2.0b2pre) Gecko/20100716 Minefield/4.0b2pre crashes pretty consistently with a clean profile.

Comment 2

[Boris Zbarsky [:bzbarsky]]

Retained layers?

Comment 3

[Matt Woodrow (:mattwoodrow)]

I can't reproduce this on a 64bit mac build. Comments in the crash report seem to indicate flash is related, which would explain it working for me.

Comment 4

[Marcia Knous [:marcia]]

I haven't been able to reproduce on a non 64 Mac build yet. So far I have only crashed on Windows.

Comment 5

[Marcia Knous [:marcia]]

http://tinyurl.com/2upoa6y confirms this is Windows only.

Comment 6

[Robert O'Callahan (:roc) (email my personal email if necessary)]

Maybe this is bug 552512 resurfacing in a new stack.

Comment 7

[Marcia Knous [:marcia]]

Crash numbers are climbing - today I see 521 crashes on Windows total since the 9th of July. The only other site that is implicated in comments is marketwatch.com, where someone wrote that they crashed while watching a video on that site.

Comment 8

[Robert O'Callahan (:roc) (email my personal email if necessary)]

I can reproduce, thanks Marcia!

Comment 9

[Robert O'Callahan (:roc) (email my personal email if necessary)]

Well, the good news is that it's not 552512.

Comment 11

[Robert O'Callahan (:roc) (email my personal email if necessary)]

This will be fixed by bug 572520, almost certainly.

Comment 12

[Mike Beltzner [:beltzner, not reading bugmail]]

Marcia notes that this is hitting top sites like Google Finance.

Comment 13

[Evgeny]

I get this crash very often (but not 100% of the time) on Google Finance, running on Windows XP x64 SP2 since installing 4.0 beta 2. Worked fine in beta 1. Eg. crash report bp-232da862-53eb-4002-ae4f-435e12100727

Comment 14

[Bobby Holley (:bholley)]

The dependent bug here just landed. Please report if you see this crash again in tomorrow's builds.

Comment 15

[Marcia Knous [:marcia]]

Using Mozilla/5.0 (Windows; Windows NT 5.1; rv:2.0b3pre) Gecko/20100729 Minefield/4.0b3pre, I was still able to generate a crash, but not on the Google finance site.

My STR for that crash:
1. Load http://www.newgrounds.com/portal/view/542858
2. Use the Dim/Restore light feature a few times.

http://crash-stats.mozilla.com/report/index/429abebb-745b-47f0-a09e-bb5912100729

Was able to crash once but have not been able to reproduce.

Comment 16

[Bobby Holley (:bholley)]

roc - looks like the above crash doesn't come from an imgIDecoderObserver event - thoughts? Might there be multiple triggers to this bug?

Comment 17

[Martijn Wargers (dead)]

I guess this is related to bug 580383 now?

Comment 18

[Marcia Knous [:marcia]]

Using Mozilla/5.0 (Windows; Windows NT 6.1; rv:2.0b3pre) Gecko/20100729 FireDownload/2.0.1 Minefield/4.0b3pre, I can crash 100% using the STR in Comment 15. I was not able to crash consistently using XP.

Comment 19

[Robert O'Callahan (:roc) (email my personal email if necessary)]

There could be multiple triggers. Basically if there's any way of entering frame construction during painting, we can crash with this bug's stack (or something very like it) now. Before we would have crashed with a different stack, one with no layer stuff and more display list stuff.

Comment 20

[chris hofmann]

now the #1 top crash on 4.0 beta 2.

Comment 21

[John]

Mozilla/5.0 (Windows; Windows NT 5.1; rv:2.0b2) Gecko/20100720 Firefox/4.0b2

If visiting a stock doesn't automatically crash you, it always crashes for me when I change the time span of a stock, like if the default is a year span, and I change to 5 years, Firefox crashes.

Comment 22

[Robert O'Callahan (:roc) (email my personal email if necessary)]

John, your build doesn't have the fix mentioned in comment #14. Google Finance should be working fine with an up-to-date build.

Comment 23

[Brendan Eich [:brendan]]

Is bug 578066 a dup?

/be

Comment 24

[Martijn Wargers (dead)]

Yes, I think so.

Comment 26

[chris hofmann]

(In reply to comment #19)
> There could be multiple triggers. Basically if there's any way of entering
> frame construction during painting, we can crash with this bug's stack (or
> something very like it) now. Before we would have crashed with a different
> stack, one with no layer stuff and more display list stuff.

The google finance crash appears to have been fixed but this still ranks high in early 40.b3 data and I suspect it will continue to be high volume as the number of users ramp up on b3.  It still could be our most visible crash regression for 4.0.

Should we figure out whats needed to construct some test cases to uncover more ways where frame construction happens during painting to help isolate? 

Should we track cases like comment 15 in other bugs or track them here?

Comment 27

[Marcia Knous [:marcia]]

chofmann asked if I could repro the crash in Comment 15 using the current trunk, and I can. http://crash-stats.mozilla.com/report/index/bp-f39fb545-b861-43ab-b408-627c82100810 is the report, which at the top of the stack looks the same as Bug 546745 which was resolved fixed a while back.

Comment 28

[Robert O'Callahan (:roc) (email my personal email if necessary)]

I was not able to reproduce with the steps in comment #15 in a Windows debug build. Marcia, can you reproduce in a debug build? If you can, you should see assertion failure alerts coming up as you try to reproduce the bug. At the first alert, choose "debug" or attach the Visual Studio debugger and get the call stack.

Comment 29

[Carsten Book [:Tomcat]]

(In reply to comment #28)
> I was not able to reproduce with the steps in comment #15 in a Windows debug
> build. Marcia, can you reproduce in a debug build? If you can, you should see
> assertion failure alerts coming up as you try to reproduce the bug. At the
> first alert, choose "debug" or attach the Visual Studio debugger and get the
> call stack.

i will check this with my debug builds and debugger and see that i can get a stack

Comment 31

[Carsten Book [:Tomcat]]

(In reply to comment #29)
> (In reply to comment #28)
> > I was not able to reproduce with the steps in comment #15 in a Windows debug
> > build. Marcia, can you reproduce in a debug build? If you can, you should see
> > assertion failure alerts coming up as you try to reproduce the bug. At the
> > first alert, choose "debug" or attach the Visual Studio debugger and get the
> > call stack.
> 
> i will check this with my debug builds and debugger and see that i can get a
> stack

so far i was not able to reproduce this crash :(

Comment 32

[Marcia Knous [:marcia]]

Tomcat is suggesting that we close this out since the Google Finance crash is fixed, and open a new bug if we can reproduce the crash in Comment 15.

Also can we can get an answer to chofmann's question on Comment 26 about tests for this bug?

Comment 33

[Robert O'Callahan (:roc) (email my personal email if necessary)]

In a debug build we assert when frame construction happens during painting. So we can find bugs like this just by testing debug builds and watching for those assertions.

I want to be able to annotate crash reports with flags to indicate when something like that happens in the field. I wrote a patch for one such case in bug 552512. But apparently we can't do that because Socorro can't handle the data.

Comment 34

[chris hofmann]

bc, sounds like we could watch for these assertions in crash test automation.

Comment 35

[Bob Clary [:bc] (inactive)]

(In reply to comment #34)
> bc, sounds like we could watch for these assertions in crash test automation.

We do track any assertion found during crash or valgrind unit testing. The difficulty is knowing which ones are interesting. The server and workers are in the process of being migrated to the phoenix colo at the moment. I hope to have them up and running again within a day or so with an increased number of workers.

If you can tell me what assertions are relevant to this bug, I'll query the db when it become available again.

Comment 36

[Bob Clary [:bc] (inactive)]

fwiw, I tested this url on winxp, and linux on 1.9.1.,1.9.2,2.0.0 and didn't see a crash or assertion.

Comment 37

[Robert O'Callahan (:roc) (email my personal email if necessary)]

Can everyone who was able to reproduce crashes reported in this bug please retest? Checkins for bug 594774 may have fixed this.

Comment 38

[Robert O'Callahan (:roc) (email my personal email if necessary)]

I guess we have to wait for beta7 to be released before confirming that this is fixed.

Comment 39

[Carsten Book [:Tomcat]]

(In reply to comment #38)
> I guess we have to wait for beta7 to be released before confirming that this is
> fixed.

ok cool thanks, will take a look then

Comment 40

[Mats Palmgren (inactive)]

Looks like it's still crashing in 4.0b8pre (buildid 20101108043306):
bp-49b6a78b-bb5c-4274-b005-73e402101109

Comment 41

[Carsten Book [:Tomcat]]

seems this is now fixed

Results within 1 weeks of 11/29/2010 11:50:53, where the crash signature is exactly 'mozilla::FrameLayerBuilder::DrawThebesLayer(mozilla::layers::ThebesLayer*, gfxContext*, nsIntRegion const&, nsIntRegion const&, void*)', and the product is one of Firefox, and the version is one of Firefox: and the crashing process was a any.

No results were found.

marking as fixed for now. Feel free to reopen when this can be reproduced with a latest build

Comment 42

[Scoobidiver (away)]

It is still there: #73 top crasher in 4.0b8pre for the last week.

http://crash-stats.mozilla.com/report/list?product=Firefox&query_search=signature&query_type=exact&query=&range_value=4&range_unit=weeks&hang_type=any&process_type=any&plugin_field=&plugin_query_type=&plugin_query=&do_query=1&admin=&signature=mozilla%3A%3AFrameLayerBuilder%3A%3ADrawThebesLayer%28mozilla%3A%3Alayers%3A%3AThebesLayer*%2C%20gfxContext*%2C%20nsIntRegion%20const%26%2C%20nsIntRegion%20const%26%2C%20void*%29

Comment 43

[Robert O'Callahan (:roc) (email my personal email if necessary)]

OK but that won't block.

Comment 44

[Scoobidiver (away)]

It happens also in Fennec 4.0b3 and is #38 top crasher.

More reports at:
http://crash-stats.mozilla.com/report/list?range_value=4&range_unit=weeks&signature=mozilla%3A%3AFrameLayerBuilder%3A%3ADrawThebesLayer

Comment 45

[Robert Kaiser]

Still happens even on trunk, I guess that status whiteboard is bogus now.

Comment 46

[Sheila Mooney]

Not a new signature. Most of the ones on the trunk yesterday seem to be tagged as dups.

Comment 47

[Sheila Mooney]

This is still a valid bug. We have 1387 of these on Firefox 8.0. It's not a top crash anymore at #164 but still probably should be looked at. Removing the top crash keyword. I am also going to remove the status whiteboard.

Comment 48

[Mats Palmgren (inactive)]

DrawThebesLayer changed name to DrawPaintedLayer
https://bugzilla.mozilla.org/attachment.cgi?id=8495457&action=diff#a/embedding/browser/nsWebBrowser.cpp_sec2

It still crash in low volume: 28 reports in the past 7 days.

Comment 49

[Andrei Purice]

Hey Marcia,
Can you still reproduce this issue?

Comment 50

[Ryan VanderMeulen [:RyanVM]]

Looks like it's still lingering around with low volume, but certainly not a critical severity issue.

Comment 51

[BugBot [:suhaib / :marco/ :calixte]]

Closing because no crashes reported for 12 weeks.