Startup 64-bit Windows PGO crash in mozilla::layers::LayerPropertiesBase::LayerPropertiesBase

VERIFIED FIXED in Firefox 18

Status

()

Core
Layout
--
blocker
VERIFIED FIXED
5 years ago
5 years ago

People

(Reporter: Scoobidiver (away), Assigned: m_kato)

Tracking

({crash, regression, topcrash})

18 Branch
mozilla19
x86_64
Windows 7
crash, regression, topcrash
Points:
---
Dependency tree / graph

Firefox Tracking Flags

(firefox18 verified)

Details

(Whiteboard: [startupcrash], crash signature)

Attachments

(1 attachment)

(Reporter)

Description

5 years ago
64-bit Nightly crashes on startup. It's #1 top crasher with about 1000 crashes an hour.
The regression range is:
http://hg.mozilla.org/mozilla-central/pushloghtml?fromchange=895f66c4eada&tochange=c09a0c022b2e
It's likely a regression from bug 539356 part 9f.
Updates should be stopped if you don't want to lose every users with 64-bit builds.

Signature 	mozilla::layers::LayerPropertiesBase::LayerPropertiesBase(mozilla::layers::Layer*) More Reports Search
UUID	926f65f3-3cde-4026-aac9-43e8d2120929
Date Processed	2012-09-29 14:00:20
Uptime	0
Last Crash	16 seconds before submission
Install Age	31 seconds since version was first installed.
Install Time	2012-09-29 13:59:38
Product	Firefox
Version	18.0a1
Build ID	20120929030624
Release Channel	nightly
OS	Windows NT
OS Version	6.1.7601 Service Pack 1
Build Architecture	amd64
Build Architecture Info	family 6 model 23 stepping 6
Crash Reason	EXCEPTION_ACCESS_VIOLATION_READ
Crash Address	0x0
App Notes 	
AdapterVendorID: 0x10de, AdapterDeviceID: 0x05e2, AdapterSubsysID: 2ae2107d, AdapterDriverVersion: 8.17.13.142
Processor Notes 	WARNING: JSON file missing Add-ons
EMCheckCompatibility	True
Adapter Vendor ID	0x10de
Adapter Device ID	0x05e2
Total Virtual Memory	8796092891136
Available Virtual Memory	8795830296576
System Memory Use Percentage	36
Available Page File	6873792512
Available Physical Memory	2731728896

Frame 	Module 	Signature 	Source
0 	xul.dll 	mozilla::layers::LayerPropertiesBase::LayerPropertiesBase 	gfx/layers/LayerTreeInvalidation.cpp:71
1 	xul.dll 	mozilla::layers::CloneLayerTreePropertiesInternal 	gfx/layers/LayerTreeInvalidation.cpp:284
2 	xul.dll 	nsDisplayList::PaintForFrame 	layout/base/nsDisplayList.cpp:1029
3 	xul.dll 	nsRegion::And 	gfx/src/nsRegion.cpp:731
4 	xul.dll 	nsDisplayText::GetBounds 	layout/generic/nsTextFrameThebes.cpp:4509
5 	xul.dll 	nsDisplayList::GetBounds 	layout/base/nsDisplayList.cpp:828
6 	xul.dll 	nsRegion::SetToElements 	gfx/src/nsRegion.cpp:296
7 	xul.dll 	mozilla::FramePropertyTable::Get 	layout/base/FramePropertyTable.cpp:74
8 	xul.dll 	nsDisplayList::PaintRoot 	layout/base/nsDisplayList.cpp:957
9 	xul.dll 	nsLayoutUtils::PaintFrame 	layout/base/nsLayoutUtils.cpp:1743

More reports at:
https://crash-stats.mozilla.com/report/list?signature=mozilla%3A%3Alayers%3A%3ALayerPropertiesBase%3A%3ALayerPropertiesBase%28mozilla%3A%3Alayers%3A%3ALayer*%29
(Reporter)

Updated

5 years ago
Duplicate of this bug: 795593
(Reporter)

Comment 2

5 years ago
There are 12.000 crashes so far.

Updated

5 years ago
tracking-firefox18: --- → ?
(Reporter)

Comment 3

5 years ago
The tracking flag is for versions that will be released. It's not the case for 64-bit builds.
tracking-firefox18: ? → ---

Updated

5 years ago
Blocks: 795643
I can't reproduce this in a --disable-optimize build.  Looks like a compiler bug.
Trying to build opt, I hit an internal compiler error.  Awesome!
There was a report on IRC that the non-pgo nightlies are not affected by this bug.
... which means that we can temporarily fix the nightlies by disabling pgo until someone can find a compiler that's capable of building win64-opt with hitting internal errors.

Where the mozconfig for the win64 nightlies live?
http://mxr.mozilla.org/mozilla-central/source/browser/config/mozconfigs/win64/nightly but that probably doesn't do you any good, because the mozconfig (which, odd though the naming is, is "nightly" meaning "nightly and on push and periodic PGO and basically everything except debug and releases") isn't where PGO gets set, http://mxr.mozilla.org/build/source/buildbotcustom/misc.py#1306 is. It's possible that the mozconfig would let you make all of on-push, periodic PGO, and nightly builds non-PGO, but I'd bet against it, and on buildbot winning instead.
Also, it seems pretty bad to be shipping nightly builds to users without running a test that the browser even starts up.
(In reply to Phil Ringnalda (:philor) from comment #8)
> http://mxr.mozilla.org/mozilla-central/source/browser/config/mozconfigs/
> win64/nightly but that probably doesn't do you any good, because the
> mozconfig (which, odd though the naming is, is "nightly" meaning "nightly
> and on push and periodic PGO and basically everything except debug and
> releases") isn't where PGO gets set,
> http://mxr.mozilla.org/build/source/buildbotcustom/misc.py#1306 is. It's
> possible that the mozconfig would let you make all of on-push, periodic PGO,
> and nightly builds non-PGO, but I'd bet against it, and on buildbot winning
> instead.

OK.  I guess this ball is in releng's court then.  Thanks for the links.
Filed bug 795668 on running tests on win64 PGO and nightly builds, not that running hidden tests will actually give us much of any warning (we could ship win32 builds that don't start up too, we just wouldn't do it for quite as long).
Duplicate of this bug: 795641
Since this is a bug in the win64 PGO, that DLBI happened to trigger, I'm unsetting the block of DLBI.
No longer blocks: 539356
Today we show 73,083 Windows crashes.
If as said in previous comments, this is PGO only, I think the correct way forward here is to not have the Nightly win 64-bit Nightlies be PGO builds.
(In reply to Bill Gianopoulos [:WG9s] from comment #15)
> If as said in previous comments, this is PGO only, I think the correct way
> forward here is to not have the Nightly win 64-bit Nightlies be PGO builds.

I meant to say until we figure this out, of course.
Summary: Startup 64-bit crash in mozilla::layers::LayerPropertiesBase::LayerPropertiesBase → Startup 64-bit Windows PGO crash in mozilla::layers::LayerPropertiesBase::LayerPropertiesBase

Updated

5 years ago
Blocks: 795748
Filed bug 795748 to resolve this startup crash in the short term.
(Assignee)

Comment 18

5 years ago
compiler generates invalid code to store float...
A non-pgo nightly is available at http://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/2012/09/2012-09-30-18-33-50-mozilla-central/ for verification that crash is gone.
(Reporter)

Updated

5 years ago
No longer blocks: 795643

Comment 20

5 years ago
Why not switch to VS 2012 for Win64 PGO build?
(In reply to Nick Thomas [:nthomas] from comment #19)
> A non-pgo nightly is available at
> http://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/2012/09/2012-09-30-18-
> 33-50-mozilla-central/ for verification that crash is gone.

This builds fixes the startup crash for me.

Comment 22

5 years ago
Confirming that this fixes the startup crash for me. Thanks!

Comment 23

5 years ago
(In reply to Yuan Pengfei from comment #20)
> Why not switch to VS 2012 for Win64 PGO build?

Because binaries produced by VS2012 are incompatible with NT5.2 (2K3 x64 and XP x64).

(In reply to Makoto Kato from comment #18)
> compiler generates invalid code to store float...

Is there a hotfix for this?  Microsoft releases hotfixes all the time to address compiler bugs, but most of them are never released via automatic updates but instead are available for download on demand by people who seek out the KB articles and then look for the fix on Microsoft Connect.

Comment 24

5 years ago
(In reply to Kai Liu from comment #23)
> (In reply to Yuan Pengfei from comment #20)
> > Why not switch to VS 2012 for Win64 PGO build?
> 
> Because binaries produced by VS2012 are incompatible with NT5.2 (2K3 x64 and
> XP x64).
Why not just drop support for these OS versions then? Microsoft's own Windows Lifecycle Fact Sheet page http://windows.microsoft.com/en-US/windows/products/lifecycle shows that mainstream support for XP ended in *2009* and Win2000 support is long gone. My guess is anyone who's bleeding edge enough to be running Nightly Win64 builds is probably running at least Windows 7 or later too.

> 
> (In reply to Makoto Kato from comment #18)
> > compiler generates invalid code to store float...
> 
> Is there a hotfix for this?  Microsoft releases hotfixes all the time to
> address compiler bugs, but most of them are never released via automatic
> updates but instead are available for download on demand by people who seek
> out the KB articles and then look for the fix on Microsoft Connect.
I'm not getting the Nightly automatic updates as usual, the latest build I have is September's 27 and checking for updates results in Nightly being up to date. I suppose this is on purpose because of the crash issue? If so, I really think it's great!

I'm using a Windows 7 64-bit PC.

Comment 26

5 years ago
(In reply to Judah Richardson from comment #24)
> (In reply to Kai Liu from comment #23)
> > (In reply to Yuan Pengfei from comment #20)
> > > Why not switch to VS 2012 for Win64 PGO build?
> > 
> > Because binaries produced by VS2012 are incompatible with NT5.2 (2K3 x64 and
> > XP x64).
> Why not just drop support for these OS versions then? Microsoft's own
> Windows Lifecycle Fact Sheet page
> http://windows.microsoft.com/en-US/windows/products/lifecycle shows that
> mainstream support for XP ended in *2009* and Win2000 support is long gone.
> My guess is anyone who's bleeding edge enough to be running Nightly Win64
> builds is probably running at least Windows 7 or later too.
> 
> > 
> > (In reply to Makoto Kato from comment #18)
> > > compiler generates invalid code to store float...
> > 
> > Is there a hotfix for this?  Microsoft releases hotfixes all the time to
> > address compiler bugs, but most of them are never released via automatic
> > updates but instead are available for download on demand by people who seek
> > out the KB articles and then look for the fix on Microsoft Connect.

I agree.
It is reasonable to drop support for WinXP x64.

Comment 27

5 years ago
(In reply to Gabriela from comment #25)
> I'm not getting the Nightly automatic updates as usual, the latest build I
> have is September's 27 and checking for updates results in Nightly being up
> to date. I suppose this is on purpose because of the crash issue? If so, I
> really think it's great!
> 
> I'm using a Windows 7 64-bit PC.

If you download the latest installer, automatic updates will work as usual. I'm using 2012-10-02 build right now.
Isn't it crashing?

Comment 29

5 years ago
(In reply to Gabriela from comment #28)
> Isn't it crashing?

If you download the non-PGO build linked to in this thread and update from that you should be fine. I suspect the update that build downloads is the same as the regular x64 update though.

Comment 30

5 years ago
(In reply to Judah Richardson from comment #29)
> (In reply to Gabriela from comment #28)
> > Isn't it crashing?
> 
> If you download the non-PGO build linked to in this thread and update from
> that you should be fine. I suspect the update that build downloads is the
> same as the regular x64 update though.

Downloading the non-PGO won't have updates, last I checked.

Comment 31

5 years ago
(In reply to Kyle Repinski from comment #30)
> (In reply to Judah Richardson from comment #29)
> > (In reply to Gabriela from comment #28)
> > > Isn't it crashing?
> > 
> > If you download the non-PGO build linked to in this thread and update from
> > that you should be fine. I suspect the update that build downloads is the
> > same as the regular x64 update though.
> 
> Downloading the non-PGO won't have updates, last I checked.

I wish there was an edit function here...

The new Nightly builds for x64 are non-PGO, they changed the build config stuff to disable PGO for them.

Comment 32

5 years ago
> I wish there was an edit function here...

I second that emotion. I'm going to file a bug.

Comment 33

5 years ago
(In reply to Joe Greenman from comment #32)
> > I wish there was an edit function here...
> 
> I second that emotion. I'm going to file a bug.

I've filed Bug 797382. Please excuse the mini-hijack.
(Assignee)

Comment 34

5 years ago
Created attachment 669863 [details] [diff] [review]
workaround fix for compiler bug
(Assignee)

Comment 35

5 years ago
Comment on attachment 669863 [details] [diff] [review]
workaround fix for compiler bug

add workaround for x64 compiler bug.  MSVC generates invalid SSE2 code.
Attachment #669863 - Flags: review?(roc)
Attachment #669863 - Flags: review?(roc) → review+
(Assignee)

Comment 36

5 years ago
https://hg.mozilla.org/integration/mozilla-inbound/rev/6c7c155557df
Assignee: nobody → m_kato
Target Milestone: --- → mozilla19

Comment 37

5 years ago
Any chance PGO can be re-enabled for win64 builds if this fixes it crashing with PGO?
https://hg.mozilla.org/mozilla-central/rev/6c7c155557df
Status: NEW → RESOLVED
Last Resolved: 5 years ago
Resolution: --- → FIXED
Blocks: 801386
Comment on attachment 669863 [details] [diff] [review]
workaround fix for compiler bug

Review of attachment 669863 [details] [diff] [review]:
-----------------------------------------------------------------

Safe patch that just disables some optimization. This will let us build Win64 PGO on Aurora again.
Attachment #669863 - Flags: approval-mozilla-aurora?

Updated

5 years ago
Attachment #669863 - Flags: approval-mozilla-aurora? → approval-mozilla-aurora+
(Assignee)

Comment 40

5 years ago
https://hg.mozilla.org/releases/mozilla-aurora/rev/04d585d5045a
status-firefox18: --- → fixed
While checking in Socorro, there can be seen crashes that appeared within the last month, including crashes on Firefox 18 beta 2.

The crash reports are available here: https://crash-stats.mozilla.com/report/list?product=Firefox&query_search=signature&query_type=contains&query=mozilla%3A%3Alayers%3A%3ALayerPropertiesBase%3A%3ALayerPropertiesBase%28mozilla%3A%3Alayers%3A%3ALayer*%29&reason_type=contains&date=12%2F05%2F2012%2008%3A07%3A48&range_value=4&range_unit=weeks&hang_type=any&process_type=any&do_query=1&signature=mozilla%3A%3Alayers%3A%3ALayerPropertiesBase%3A%3ALayerPropertiesBase%28mozilla%3A%3Alayers%3A%3ALayer*%29

Updated

5 years ago
QA Contact: manuela.muntean

Comment 42

5 years ago
(In reply to Manuela Muntean from comment #41)
> While checking in Socorro, there can be seen crashes that appeared within
> the last month, including crashes on Firefox 18 beta 2.

Those are pretty surely something else that happens to end up with the same signature, as those are all 32bit builds crashing while this bug is about a 64bit problem.
(In reply to Robert Kaiser (:kairo@mozilla.com) from comment #42)
> (In reply to Manuela Muntean from comment #41)
> > While checking in Socorro, there can be seen crashes that appeared within
> > the last month, including crashes on Firefox 18 beta 2.
> 
> Those are pretty surely something else that happens to end up with the same
> signature, as those are all 32bit builds crashing while this bug is about a
> 64bit problem.

Thanks Robert for your reply. Based on this afirmation I'm marking this fixed.
Status: RESOLVED → VERIFIED
status-firefox18: fixed → verified
You need to log in before you can comment on or make changes to this bug.