795594 - Startup 64-bit Windows PGO crash in mozilla::layers::LayerPropertiesBase::LayerPropertiesBase

Status: VERIFIED FIXED

(Core :: Layout, defect)

Description

[Scoobidiver (away)]

64-bit Nightly crashes on startup. It's #1 top crasher with about 1000 crashes an hour.
The regression range is:
http://hg.mozilla.org/mozilla-central/pushloghtml?fromchange=895f66c4eada&tochange=c09a0c022b2e
It's likely a regression from bug 539356 part 9f.
Updates should be stopped if you don't want to lose every users with 64-bit builds.

Signature 	mozilla::layers::LayerPropertiesBase::LayerPropertiesBase(mozilla::layers::Layer*) More Reports Search
UUID	926f65f3-3cde-4026-aac9-43e8d2120929
Date Processed	2012-09-29 14:00:20
Uptime	0
Last Crash	16 seconds before submission
Install Age	31 seconds since version was first installed.
Install Time	2012-09-29 13:59:38
Product	Firefox
Version	18.0a1
Build ID	20120929030624
Release Channel	nightly
OS	Windows NT
OS Version	6.1.7601 Service Pack 1
Build Architecture	amd64
Build Architecture Info	family 6 model 23 stepping 6
Crash Reason	EXCEPTION_ACCESS_VIOLATION_READ
Crash Address	0x0
App Notes 	
AdapterVendorID: 0x10de, AdapterDeviceID: 0x05e2, AdapterSubsysID: 2ae2107d, AdapterDriverVersion: 8.17.13.142
Processor Notes 	WARNING: JSON file missing Add-ons
EMCheckCompatibility	True
Adapter Vendor ID	0x10de
Adapter Device ID	0x05e2
Total Virtual Memory	8796092891136
Available Virtual Memory	8795830296576
System Memory Use Percentage	36
Available Page File	6873792512
Available Physical Memory	2731728896

Frame 	Module 	Signature 	Source
0 	xul.dll 	mozilla::layers::LayerPropertiesBase::LayerPropertiesBase 	gfx/layers/LayerTreeInvalidation.cpp:71
1 	xul.dll 	mozilla::layers::CloneLayerTreePropertiesInternal 	gfx/layers/LayerTreeInvalidation.cpp:284
2 	xul.dll 	nsDisplayList::PaintForFrame 	layout/base/nsDisplayList.cpp:1029
3 	xul.dll 	nsRegion::And 	gfx/src/nsRegion.cpp:731
4 	xul.dll 	nsDisplayText::GetBounds 	layout/generic/nsTextFrameThebes.cpp:4509
5 	xul.dll 	nsDisplayList::GetBounds 	layout/base/nsDisplayList.cpp:828
6 	xul.dll 	nsRegion::SetToElements 	gfx/src/nsRegion.cpp:296
7 	xul.dll 	mozilla::FramePropertyTable::Get 	layout/base/FramePropertyTable.cpp:74
8 	xul.dll 	nsDisplayList::PaintRoot 	layout/base/nsDisplayList.cpp:957
9 	xul.dll 	nsLayoutUtils::PaintFrame 	layout/base/nsLayoutUtils.cpp:1743

More reports at:
https://crash-stats.mozilla.com/report/list?signature=mozilla%3A%3Alayers%3A%3ALayerPropertiesBase%3A%3ALayerPropertiesBase%28mozilla%3A%3Alayers%3A%3ALayer*%29

Comment 2

[Scoobidiver (away)]

There are 12.000 crashes so far.

Comment 3

[Scoobidiver (away)]

The tracking flag is for versions that will be released. It's not the case for 64-bit builds.

Comment 4

[Chris Jones [:cjones] inactive; ni?/f?/r? if you need me]

I can't reproduce this in a --disable-optimize build.  Looks like a compiler bug.

Comment 5

[Chris Jones [:cjones] inactive; ni?/f?/r? if you need me]

Trying to build opt, I hit an internal compiler error.  Awesome!

Comment 6

[Chris Jones [:cjones] inactive; ni?/f?/r? if you need me]

There was a report on IRC that the non-pgo nightlies are not affected by this bug.

Comment 7

[Chris Jones [:cjones] inactive; ni?/f?/r? if you need me]

... which means that we can temporarily fix the nightlies by disabling pgo until someone can find a compiler that's capable of building win64-opt with hitting internal errors.

Where the mozconfig for the win64 nightlies live?

Comment 8

[Phil Ringnalda (:philor)]

http://mxr.mozilla.org/mozilla-central/source/browser/config/mozconfigs/win64/nightly but that probably doesn't do you any good, because the mozconfig (which, odd though the naming is, is "nightly" meaning "nightly and on push and periodic PGO and basically everything except debug and releases") isn't where PGO gets set, http://mxr.mozilla.org/build/source/buildbotcustom/misc.py#1306 is. It's possible that the mozconfig would let you make all of on-push, periodic PGO, and nightly builds non-PGO, but I'd bet against it, and on buildbot winning instead.

Comment 9

[Chris Jones [:cjones] inactive; ni?/f?/r? if you need me]

Also, it seems pretty bad to be shipping nightly builds to users without running a test that the browser even starts up.

Comment 10

[Chris Jones [:cjones] inactive; ni?/f?/r? if you need me]

(In reply to Phil Ringnalda (:philor) from comment #8)
> http://mxr.mozilla.org/mozilla-central/source/browser/config/mozconfigs/
> win64/nightly but that probably doesn't do you any good, because the
> mozconfig (which, odd though the naming is, is "nightly" meaning "nightly
> and on push and periodic PGO and basically everything except debug and
> releases") isn't where PGO gets set,
> http://mxr.mozilla.org/build/source/buildbotcustom/misc.py#1306 is. It's
> possible that the mozconfig would let you make all of on-push, periodic PGO,
> and nightly builds non-PGO, but I'd bet against it, and on buildbot winning
> instead.

OK.  I guess this ball is in releng's court then.  Thanks for the links.

Comment 11

[Phil Ringnalda (:philor)]

Filed bug 795668 on running tests on win64 PGO and nightly builds, not that running hidden tests will actually give us much of any warning (we could ship win32 builds that don't start up too, we just wouldn't do it for quite as long).

Comment 13

[Chris Jones [:cjones] inactive; ni?/f?/r? if you need me]

Since this is a bug in the win64 PGO, that DLBI happened to trigger, I'm unsetting the block of DLBI.

Comment 14

[Marcia Knous [:marcia]]

Today we show 73,083 Windows crashes.

Comment 15

[Bill Gianopoulos [:WG9s]]

If as said in previous comments, this is PGO only, I think the correct way forward here is to not have the Nightly win 64-bit Nightlies be PGO builds.

Comment 16

[Bill Gianopoulos [:WG9s]]

(In reply to Bill Gianopoulos [:WG9s] from comment #15)
> If as said in previous comments, this is PGO only, I think the correct way
> forward here is to not have the Nightly win 64-bit Nightlies be PGO builds.

I meant to say until we figure this out, of course.

Comment 17

[Alex Keybl [:akeybl]]

Filed bug 795748 to resolve this startup crash in the short term.

Comment 18

[Makoto Kato [:m_kato]]

compiler generates invalid code to store float...

Comment 19

[Nick Thomas [:nthomas] (UTC+12)]

A non-pgo nightly is available at http://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/2012/09/2012-09-30-18-33-50-mozilla-central/ for verification that crash is gone.

Comment 20

[Yuan Pengfei]

Why not switch to VS 2012 for Win64 PGO build?

Comment 21

[Bill Gianopoulos [:WG9s]]

(In reply to Nick Thomas [:nthomas] from comment #19)
> A non-pgo nightly is available at
> http://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/2012/09/2012-09-30-18-
> 33-50-mozilla-central/ for verification that crash is gone.

This builds fixes the startup crash for me.

Comment 22

[Judah Richardson]

Confirming that this fixes the startup crash for me. Thanks!

Comment 23

[Kai Liu]

(In reply to Yuan Pengfei from comment #20)
> Why not switch to VS 2012 for Win64 PGO build?

Because binaries produced by VS2012 are incompatible with NT5.2 (2K3 x64 and XP x64).

(In reply to Makoto Kato from comment #18)
> compiler generates invalid code to store float...

Is there a hotfix for this?  Microsoft releases hotfixes all the time to address compiler bugs, but most of them are never released via automatic updates but instead are available for download on demand by people who seek out the KB articles and then look for the fix on Microsoft Connect.

Comment 24

[Judah Richardson]

(In reply to Kai Liu from comment #23)
> (In reply to Yuan Pengfei from comment #20)
> > Why not switch to VS 2012 for Win64 PGO build?
> 
> Because binaries produced by VS2012 are incompatible with NT5.2 (2K3 x64 and
> XP x64).
Why not just drop support for these OS versions then? Microsoft's own Windows Lifecycle Fact Sheet page http://windows.microsoft.com/en-US/windows/products/lifecycle shows that mainstream support for XP ended in *2009* and Win2000 support is long gone. My guess is anyone who's bleeding edge enough to be running Nightly Win64 builds is probably running at least Windows 7 or later too.

> 
> (In reply to Makoto Kato from comment #18)
> > compiler generates invalid code to store float...
> 
> Is there a hotfix for this?  Microsoft releases hotfixes all the time to
> address compiler bugs, but most of them are never released via automatic
> updates but instead are available for download on demand by people who seek
> out the KB articles and then look for the fix on Microsoft Connect.

Comment 25

[Gabriela [:gaby2300]]

I'm not getting the Nightly automatic updates as usual, the latest build I have is September's 27 and checking for updates results in Nightly being up to date. I suppose this is on purpose because of the crash issue? If so, I really think it's great!

I'm using a Windows 7 64-bit PC.

Comment 26

[Yuan Pengfei]

(In reply to Judah Richardson from comment #24)
> (In reply to Kai Liu from comment #23)
> > (In reply to Yuan Pengfei from comment #20)
> > > Why not switch to VS 2012 for Win64 PGO build?
> > 
> > Because binaries produced by VS2012 are incompatible with NT5.2 (2K3 x64 and
> > XP x64).
> Why not just drop support for these OS versions then? Microsoft's own
> Windows Lifecycle Fact Sheet page
> http://windows.microsoft.com/en-US/windows/products/lifecycle shows that
> mainstream support for XP ended in *2009* and Win2000 support is long gone.
> My guess is anyone who's bleeding edge enough to be running Nightly Win64
> builds is probably running at least Windows 7 or later too.
> 
> > 
> > (In reply to Makoto Kato from comment #18)
> > > compiler generates invalid code to store float...
> > 
> > Is there a hotfix for this?  Microsoft releases hotfixes all the time to
> > address compiler bugs, but most of them are never released via automatic
> > updates but instead are available for download on demand by people who seek
> > out the KB articles and then look for the fix on Microsoft Connect.

I agree.
It is reasonable to drop support for WinXP x64.

Comment 27

[Kyle Repinski]

(In reply to Gabriela from comment #25)
> I'm not getting the Nightly automatic updates as usual, the latest build I
> have is September's 27 and checking for updates results in Nightly being up
> to date. I suppose this is on purpose because of the crash issue? If so, I
> really think it's great!
> 
> I'm using a Windows 7 64-bit PC.

If you download the latest installer, automatic updates will work as usual. I'm using 2012-10-02 build right now.

Comment 28

[Gabriela [:gaby2300]]

Isn't it crashing?

Comment 29

[Judah Richardson]

(In reply to Gabriela from comment #28)
> Isn't it crashing?

If you download the non-PGO build linked to in this thread and update from that you should be fine. I suspect the update that build downloads is the same as the regular x64 update though.

Comment 30

[Kyle Repinski]

(In reply to Judah Richardson from comment #29)
> (In reply to Gabriela from comment #28)
> > Isn't it crashing?
> 
> If you download the non-PGO build linked to in this thread and update from
> that you should be fine. I suspect the update that build downloads is the
> same as the regular x64 update though.

Downloading the non-PGO won't have updates, last I checked.

Comment 31

[Kyle Repinski]

(In reply to Kyle Repinski from comment #30)
> (In reply to Judah Richardson from comment #29)
> > (In reply to Gabriela from comment #28)
> > > Isn't it crashing?
> > 
> > If you download the non-PGO build linked to in this thread and update from
> > that you should be fine. I suspect the update that build downloads is the
> > same as the regular x64 update though.
> 
> Downloading the non-PGO won't have updates, last I checked.

I wish there was an edit function here...

The new Nightly builds for x64 are non-PGO, they changed the build config stuff to disable PGO for them.

Comment 32

[u52928]

> I wish there was an edit function here...

I second that emotion. I'm going to file a bug.

Comment 33

[u52928]

(In reply to Joe Greenman from comment #32)
> > I wish there was an edit function here...
> 
> I second that emotion. I'm going to file a bug.

I've filed Bug 797382. Please excuse the mini-hijack.

Comment 34

[Makoto Kato [:m_kato]]

Attachment: workaround fix for compiler bug

Comment 35

[Makoto Kato [:m_kato]]

Comment on attachment 669863 [details] [diff] [review]
workaround fix for compiler bug

add workaround for x64 compiler bug.  MSVC generates invalid SSE2 code.

Comment 36

[Makoto Kato [:m_kato]]

https://hg.mozilla.org/integration/mozilla-inbound/rev/6c7c155557df

Comment 37

[Kyle Repinski]

Any chance PGO can be re-enabled for win64 builds if this fixes it crashing with PGO?

Comment 38

[Ed Morley [:emorley]]

https://hg.mozilla.org/mozilla-central/rev/6c7c155557df

Comment 39

[Robert O'Callahan (:roc) (email my personal email if necessary)]

Comment on attachment 669863 [details] [diff] [review]
workaround fix for compiler bug

Review of attachment 669863 [details] [diff] [review]:
-----------------------------------------------------------------

Safe patch that just disables some optimization. This will let us build Win64 PGO on Aurora again.

Comment 40

[Makoto Kato [:m_kato]]

https://hg.mozilla.org/releases/mozilla-aurora/rev/04d585d5045a

Comment 41

[Manuela Muntean [Away]]

While checking in Socorro, there can be seen crashes that appeared within the last month, including crashes on Firefox 18 beta 2.

The crash reports are available here: https://crash-stats.mozilla.com/report/list?product=Firefox&query_search=signature&query_type=contains&query=mozilla%3A%3Alayers%3A%3ALayerPropertiesBase%3A%3ALayerPropertiesBase%28mozilla%3A%3Alayers%3A%3ALayer*%29&reason_type=contains&date=12%2F05%2F2012%2008%3A07%3A48&range_value=4&range_unit=weeks&hang_type=any&process_type=any&do_query=1&signature=mozilla%3A%3Alayers%3A%3ALayerPropertiesBase%3A%3ALayerPropertiesBase%28mozilla%3A%3Alayers%3A%3ALayer*%29

Comment 42

[Robert Kaiser]

(In reply to Manuela Muntean from comment #41)
> While checking in Socorro, there can be seen crashes that appeared within
> the last month, including crashes on Firefox 18 beta 2.

Those are pretty surely something else that happens to end up with the same signature, as those are all 32bit builds crashing while this bug is about a 64bit problem.

Comment 43

[Manuela Muntean [Away]]

(In reply to Robert Kaiser (:[e-mail]) from comment #42)
> (In reply to Manuela Muntean from comment #41)
> > While checking in Socorro, there can be seen crashes that appeared within
> > the last month, including crashes on Firefox 18 beta 2.
> 
> Those are pretty surely something else that happens to end up with the same
> signature, as those are all 32bit builds crashing while this bug is about a
> 64bit problem.

Thanks Robert for your reply. Based on this afirmation I'm marking this fixed.