The default bug view has changed. See this FAQ.
Last Comment Bug 795594 - Startup 64-bit Windows PGO crash in mozilla::layers::LayerPropertiesBase::LayerPropertiesBase
: Startup 64-bit Windows PGO crash in mozilla::layers::LayerPropertiesBase::Lay...
Status: VERIFIED FIXED
[startupcrash]
: crash, regression, topcrash
Product: Core
Classification: Components
Component: Layout (show other bugs)
: 18 Branch
: x86_64 Windows 7
-- blocker with 11 votes (vote)
: mozilla19
Assigned To: Makoto Kato [:m_kato]
: Manuela Muntean [Away]
: Jet Villegas (:jet)
Mentors:
: 795593 795641 (view as bug list)
Depends on:
Blocks: 795748 801386
  Show dependency treegraph
 
Reported: 2012-09-29 07:04 PDT by Scoobidiver (away)
Modified: 2012-12-05 06:09 PST (History)
35 users (show)
See Also:
Crash Signature:
QA Whiteboard:
Iteration: ---
Points: ---
Has Regression Range: ---
Has STR: ---
verified


Attachments
workaround fix for compiler bug (653 bytes, patch)
2012-10-09 22:05 PDT, Makoto Kato [:m_kato]
roc: review+
bajaj.bhavana: approval‑mozilla‑aurora+
Details | Diff | Splinter Review

Description User image Scoobidiver (away) 2012-09-29 07:04:46 PDT
64-bit Nightly crashes on startup. It's #1 top crasher with about 1000 crashes an hour.
The regression range is:
http://hg.mozilla.org/mozilla-central/pushloghtml?fromchange=895f66c4eada&tochange=c09a0c022b2e
It's likely a regression from bug 539356 part 9f.
Updates should be stopped if you don't want to lose every users with 64-bit builds.

Signature 	mozilla::layers::LayerPropertiesBase::LayerPropertiesBase(mozilla::layers::Layer*) More Reports Search
UUID	926f65f3-3cde-4026-aac9-43e8d2120929
Date Processed	2012-09-29 14:00:20
Uptime	0
Last Crash	16 seconds before submission
Install Age	31 seconds since version was first installed.
Install Time	2012-09-29 13:59:38
Product	Firefox
Version	18.0a1
Build ID	20120929030624
Release Channel	nightly
OS	Windows NT
OS Version	6.1.7601 Service Pack 1
Build Architecture	amd64
Build Architecture Info	family 6 model 23 stepping 6
Crash Reason	EXCEPTION_ACCESS_VIOLATION_READ
Crash Address	0x0
App Notes 	
AdapterVendorID: 0x10de, AdapterDeviceID: 0x05e2, AdapterSubsysID: 2ae2107d, AdapterDriverVersion: 8.17.13.142
Processor Notes 	WARNING: JSON file missing Add-ons
EMCheckCompatibility	True
Adapter Vendor ID	0x10de
Adapter Device ID	0x05e2
Total Virtual Memory	8796092891136
Available Virtual Memory	8795830296576
System Memory Use Percentage	36
Available Page File	6873792512
Available Physical Memory	2731728896

Frame 	Module 	Signature 	Source
0 	xul.dll 	mozilla::layers::LayerPropertiesBase::LayerPropertiesBase 	gfx/layers/LayerTreeInvalidation.cpp:71
1 	xul.dll 	mozilla::layers::CloneLayerTreePropertiesInternal 	gfx/layers/LayerTreeInvalidation.cpp:284
2 	xul.dll 	nsDisplayList::PaintForFrame 	layout/base/nsDisplayList.cpp:1029
3 	xul.dll 	nsRegion::And 	gfx/src/nsRegion.cpp:731
4 	xul.dll 	nsDisplayText::GetBounds 	layout/generic/nsTextFrameThebes.cpp:4509
5 	xul.dll 	nsDisplayList::GetBounds 	layout/base/nsDisplayList.cpp:828
6 	xul.dll 	nsRegion::SetToElements 	gfx/src/nsRegion.cpp:296
7 	xul.dll 	mozilla::FramePropertyTable::Get 	layout/base/FramePropertyTable.cpp:74
8 	xul.dll 	nsDisplayList::PaintRoot 	layout/base/nsDisplayList.cpp:957
9 	xul.dll 	nsLayoutUtils::PaintFrame 	layout/base/nsLayoutUtils.cpp:1743

More reports at:
https://crash-stats.mozilla.com/report/list?signature=mozilla%3A%3Alayers%3A%3ALayerPropertiesBase%3A%3ALayerPropertiesBase%28mozilla%3A%3Alayers%3A%3ALayer*%29
Comment 1 User image Scoobidiver (away) 2012-09-29 07:07:42 PDT
*** Bug 795593 has been marked as a duplicate of this bug. ***
Comment 2 User image Scoobidiver (away) 2012-09-29 09:42:26 PDT
There are 12.000 crashes so far.
Comment 3 User image Scoobidiver (away) 2012-09-29 11:27:20 PDT
The tracking flag is for versions that will be released. It's not the case for 64-bit builds.
Comment 4 User image Chris Jones [:cjones] inactive; ni?/f?/r? if you need me 2012-09-29 19:52:26 PDT
I can't reproduce this in a --disable-optimize build.  Looks like a compiler bug.
Comment 5 User image Chris Jones [:cjones] inactive; ni?/f?/r? if you need me 2012-09-29 20:58:58 PDT
Trying to build opt, I hit an internal compiler error.  Awesome!
Comment 6 User image Chris Jones [:cjones] inactive; ni?/f?/r? if you need me 2012-09-29 20:59:46 PDT
There was a report on IRC that the non-pgo nightlies are not affected by this bug.
Comment 7 User image Chris Jones [:cjones] inactive; ni?/f?/r? if you need me 2012-09-29 21:06:59 PDT
... which means that we can temporarily fix the nightlies by disabling pgo until someone can find a compiler that's capable of building win64-opt with hitting internal errors.

Where the mozconfig for the win64 nightlies live?
Comment 8 User image Phil Ringnalda (:philor) 2012-09-29 21:19:37 PDT
http://mxr.mozilla.org/mozilla-central/source/browser/config/mozconfigs/win64/nightly but that probably doesn't do you any good, because the mozconfig (which, odd though the naming is, is "nightly" meaning "nightly and on push and periodic PGO and basically everything except debug and releases") isn't where PGO gets set, http://mxr.mozilla.org/build/source/buildbotcustom/misc.py#1306 is. It's possible that the mozconfig would let you make all of on-push, periodic PGO, and nightly builds non-PGO, but I'd bet against it, and on buildbot winning instead.
Comment 9 User image Chris Jones [:cjones] inactive; ni?/f?/r? if you need me 2012-09-29 21:48:56 PDT
Also, it seems pretty bad to be shipping nightly builds to users without running a test that the browser even starts up.
Comment 10 User image Chris Jones [:cjones] inactive; ni?/f?/r? if you need me 2012-09-29 21:49:43 PDT
(In reply to Phil Ringnalda (:philor) from comment #8)
> http://mxr.mozilla.org/mozilla-central/source/browser/config/mozconfigs/
> win64/nightly but that probably doesn't do you any good, because the
> mozconfig (which, odd though the naming is, is "nightly" meaning "nightly
> and on push and periodic PGO and basically everything except debug and
> releases") isn't where PGO gets set,
> http://mxr.mozilla.org/build/source/buildbotcustom/misc.py#1306 is. It's
> possible that the mozconfig would let you make all of on-push, periodic PGO,
> and nightly builds non-PGO, but I'd bet against it, and on buildbot winning
> instead.

OK.  I guess this ball is in releng's court then.  Thanks for the links.
Comment 11 User image Phil Ringnalda (:philor) 2012-09-29 22:50:15 PDT
Filed bug 795668 on running tests on win64 PGO and nightly builds, not that running hidden tests will actually give us much of any warning (we could ship win32 builds that don't start up too, we just wouldn't do it for quite as long).
Comment 12 User image Phil Ringnalda (:philor) 2012-09-29 23:11:16 PDT
*** Bug 795641 has been marked as a duplicate of this bug. ***
Comment 13 User image Chris Jones [:cjones] inactive; ni?/f?/r? if you need me 2012-09-30 01:50:53 PDT
Since this is a bug in the win64 PGO, that DLBI happened to trigger, I'm unsetting the block of DLBI.
Comment 14 User image Marcia Knous [:marcia - use ni] 2012-09-30 08:19:16 PDT
Today we show 73,083 Windows crashes.
Comment 15 User image Bill Gianopoulos [:WG9s] 2012-09-30 10:57:18 PDT
If as said in previous comments, this is PGO only, I think the correct way forward here is to not have the Nightly win 64-bit Nightlies be PGO builds.
Comment 16 User image Bill Gianopoulos [:WG9s] 2012-09-30 11:00:02 PDT
(In reply to Bill Gianopoulos [:WG9s] from comment #15)
> If as said in previous comments, this is PGO only, I think the correct way
> forward here is to not have the Nightly win 64-bit Nightlies be PGO builds.

I meant to say until we figure this out, of course.
Comment 17 User image Alex Keybl [:akeybl] 2012-09-30 14:04:49 PDT
Filed bug 795748 to resolve this startup crash in the short term.
Comment 18 User image Makoto Kato [:m_kato] 2012-09-30 18:15:20 PDT
compiler generates invalid code to store float...
Comment 19 User image Nick Thomas [:nthomas] 2012-09-30 19:43:56 PDT
A non-pgo nightly is available at http://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/2012/09/2012-09-30-18-33-50-mozilla-central/ for verification that crash is gone.
Comment 20 User image Yuan Pengfei 2012-10-01 03:28:33 PDT
Why not switch to VS 2012 for Win64 PGO build?
Comment 21 User image Bill Gianopoulos [:WG9s] 2012-10-01 05:01:31 PDT
(In reply to Nick Thomas [:nthomas] from comment #19)
> A non-pgo nightly is available at
> http://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/2012/09/2012-09-30-18-
> 33-50-mozilla-central/ for verification that crash is gone.

This builds fixes the startup crash for me.
Comment 22 User image Judah Richardson 2012-10-01 07:00:35 PDT
Confirming that this fixes the startup crash for me. Thanks!
Comment 23 User image Kai Liu 2012-10-01 08:34:47 PDT
(In reply to Yuan Pengfei from comment #20)
> Why not switch to VS 2012 for Win64 PGO build?

Because binaries produced by VS2012 are incompatible with NT5.2 (2K3 x64 and XP x64).

(In reply to Makoto Kato from comment #18)
> compiler generates invalid code to store float...

Is there a hotfix for this?  Microsoft releases hotfixes all the time to address compiler bugs, but most of them are never released via automatic updates but instead are available for download on demand by people who seek out the KB articles and then look for the fix on Microsoft Connect.
Comment 24 User image Judah Richardson 2012-10-01 08:40:53 PDT
(In reply to Kai Liu from comment #23)
> (In reply to Yuan Pengfei from comment #20)
> > Why not switch to VS 2012 for Win64 PGO build?
> 
> Because binaries produced by VS2012 are incompatible with NT5.2 (2K3 x64 and
> XP x64).
Why not just drop support for these OS versions then? Microsoft's own Windows Lifecycle Fact Sheet page http://windows.microsoft.com/en-US/windows/products/lifecycle shows that mainstream support for XP ended in *2009* and Win2000 support is long gone. My guess is anyone who's bleeding edge enough to be running Nightly Win64 builds is probably running at least Windows 7 or later too.

> 
> (In reply to Makoto Kato from comment #18)
> > compiler generates invalid code to store float...
> 
> Is there a hotfix for this?  Microsoft releases hotfixes all the time to
> address compiler bugs, but most of them are never released via automatic
> updates but instead are available for download on demand by people who seek
> out the KB articles and then look for the fix on Microsoft Connect.
Comment 25 User image Gabriela [:gaby2300] 2012-10-01 12:33:50 PDT
I'm not getting the Nightly automatic updates as usual, the latest build I have is September's 27 and checking for updates results in Nightly being up to date. I suppose this is on purpose because of the crash issue? If so, I really think it's great!

I'm using a Windows 7 64-bit PC.
Comment 26 User image Yuan Pengfei 2012-10-01 16:35:09 PDT
(In reply to Judah Richardson from comment #24)
> (In reply to Kai Liu from comment #23)
> > (In reply to Yuan Pengfei from comment #20)
> > > Why not switch to VS 2012 for Win64 PGO build?
> > 
> > Because binaries produced by VS2012 are incompatible with NT5.2 (2K3 x64 and
> > XP x64).
> Why not just drop support for these OS versions then? Microsoft's own
> Windows Lifecycle Fact Sheet page
> http://windows.microsoft.com/en-US/windows/products/lifecycle shows that
> mainstream support for XP ended in *2009* and Win2000 support is long gone.
> My guess is anyone who's bleeding edge enough to be running Nightly Win64
> builds is probably running at least Windows 7 or later too.
> 
> > 
> > (In reply to Makoto Kato from comment #18)
> > > compiler generates invalid code to store float...
> > 
> > Is there a hotfix for this?  Microsoft releases hotfixes all the time to
> > address compiler bugs, but most of them are never released via automatic
> > updates but instead are available for download on demand by people who seek
> > out the KB articles and then look for the fix on Microsoft Connect.

I agree.
It is reasonable to drop support for WinXP x64.
Comment 27 User image Kyle Repinski 2012-10-02 07:59:18 PDT
(In reply to Gabriela from comment #25)
> I'm not getting the Nightly automatic updates as usual, the latest build I
> have is September's 27 and checking for updates results in Nightly being up
> to date. I suppose this is on purpose because of the crash issue? If so, I
> really think it's great!
> 
> I'm using a Windows 7 64-bit PC.

If you download the latest installer, automatic updates will work as usual. I'm using 2012-10-02 build right now.
Comment 28 User image Gabriela [:gaby2300] 2012-10-02 08:06:00 PDT
Isn't it crashing?
Comment 29 User image Judah Richardson 2012-10-02 08:08:04 PDT
(In reply to Gabriela from comment #28)
> Isn't it crashing?

If you download the non-PGO build linked to in this thread and update from that you should be fine. I suspect the update that build downloads is the same as the regular x64 update though.
Comment 30 User image Kyle Repinski 2012-10-02 16:59:51 PDT
(In reply to Judah Richardson from comment #29)
> (In reply to Gabriela from comment #28)
> > Isn't it crashing?
> 
> If you download the non-PGO build linked to in this thread and update from
> that you should be fine. I suspect the update that build downloads is the
> same as the regular x64 update though.

Downloading the non-PGO won't have updates, last I checked.
Comment 31 User image Kyle Repinski 2012-10-02 17:01:19 PDT
(In reply to Kyle Repinski from comment #30)
> (In reply to Judah Richardson from comment #29)
> > (In reply to Gabriela from comment #28)
> > > Isn't it crashing?
> > 
> > If you download the non-PGO build linked to in this thread and update from
> > that you should be fine. I suspect the update that build downloads is the
> > same as the regular x64 update though.
> 
> Downloading the non-PGO won't have updates, last I checked.

I wish there was an edit function here...

The new Nightly builds for x64 are non-PGO, they changed the build config stuff to disable PGO for them.
Comment 32 User image JoeG 2012-10-03 08:38:07 PDT
> I wish there was an edit function here...

I second that emotion. I'm going to file a bug.
Comment 33 User image JoeG 2012-10-03 09:15:58 PDT
(In reply to Joe Greenman from comment #32)
> > I wish there was an edit function here...
> 
> I second that emotion. I'm going to file a bug.

I've filed Bug 797382. Please excuse the mini-hijack.
Comment 34 User image Makoto Kato [:m_kato] 2012-10-09 22:05:21 PDT
Created attachment 669863 [details] [diff] [review]
workaround fix for compiler bug
Comment 35 User image Makoto Kato [:m_kato] 2012-10-09 22:07:38 PDT
Comment on attachment 669863 [details] [diff] [review]
workaround fix for compiler bug

add workaround for x64 compiler bug.  MSVC generates invalid SSE2 code.
Comment 37 User image Kyle Repinski 2012-10-10 10:37:05 PDT
Any chance PGO can be re-enabled for win64 builds if this fixes it crashing with PGO?
Comment 38 User image Ed Morley (Away 22nd-27th, 29th-2nd) [:emorley] 2012-10-11 07:05:19 PDT
https://hg.mozilla.org/mozilla-central/rev/6c7c155557df
Comment 39 User image Robert O'Callahan (:roc) (email my personal email if necessary) 2012-10-14 14:39:50 PDT
Comment on attachment 669863 [details] [diff] [review]
workaround fix for compiler bug

Review of attachment 669863 [details] [diff] [review]:
-----------------------------------------------------------------

Safe patch that just disables some optimization. This will let us build Win64 PGO on Aurora again.
Comment 42 User image Robert Kaiser 2012-12-05 06:07:17 PST
(In reply to Manuela Muntean from comment #41)
> While checking in Socorro, there can be seen crashes that appeared within
> the last month, including crashes on Firefox 18 beta 2.

Those are pretty surely something else that happens to end up with the same signature, as those are all 32bit builds crashing while this bug is about a 64bit problem.
Comment 43 User image Manuela Muntean [Away] 2012-12-05 06:09:21 PST
(In reply to Robert Kaiser (:kairo@mozilla.com) from comment #42)
> (In reply to Manuela Muntean from comment #41)
> > While checking in Socorro, there can be seen crashes that appeared within
> > the last month, including crashes on Firefox 18 beta 2.
> 
> Those are pretty surely something else that happens to end up with the same
> signature, as those are all 32bit builds crashing while this bug is about a
> 64bit problem.

Thanks Robert for your reply. Based on this afirmation I'm marking this fixed.

Note You need to log in before you can comment on or make changes to this bug.