1275030 - Perma startup crash when using the skia content backend on Windows 7

Status: RESOLVED DUPLICATE of bug 1046205

(Core :: Graphics, defect)

Description

[Mason Chang [Inactive] [:mchang]]

I can basically perma stat up crash when enabling the skia content backend on Windows 7 only. My windows 10 machine doesn't seem to have this problem.

STR:
1) go to about:config
2) Change preference "gfx.content.azure.backends" to "skia"
3) Restart and perma-crash

This happened on nightly between 05/19/2016 and 05/20/2016. However, I can't actually bisect this locally as local builds do not reproduce the problem, only official nightly builds.

Crash report - https://crash-stats.mozilla.com/report/index/ffe89410-fae0-4733-8cfc-35c1d2160523

Comment 1

[Mason Chang [Inactive] [:mchang]]

I can only reproduce this with a 32 bit build.

Comment 2

[Mason Chang [Inactive] [:mchang]]

I tried going through mozregression for both Opt / PGO builds, and even using the same profile, I could not reproduce the permacrash through mozregression.

Comment 3

[Mason Chang [Inactive] [:mchang]]

Interestingly, if I run Nightly from a downloaded .zip file from ftp.mozilla.org, everything works. If I install nightly via the installer, then I get the crash.

Comment 4

[Mason Chang [Inactive] [:mchang]]

I narrowed the installation range. The first bad nightly is April 27 [1]. If I download the zip [2], everything works fine, except I get a popup saying "The patch could not be applied". 

Ryan, do you know who owns the installer or what changes with the installer? Thanks!

[1] https://ftp.mozilla.org/pub/firefox/nightly/2016/04/2016-04-27-03-02-15-mozilla-central/firefox-49.0a1.en-US.win32.installer.exe
[2] https://ftp.mozilla.org/pub/firefox/nightly/2016/04/2016-04-27-03-02-15-mozilla-central/firefox-49.0a1.en-US.win32.zip

Comment 5

[Ryan VanderMeulen [:RyanVM]]

Rob Strong or Stephen Pohl. That said, here's a regression range between that nightly and the previous one if you want to start looking for possible culprits:
https://hg.mozilla.org/mozilla-central/pushloghtml?fromchange=9ce31e9f90cb&tochange=ab0044bfa1df858919797bcd6a9aef76a668cd4a

I can't say anything stands out there :\. Maybe try grabbing some of those builds from fx-team/inbound and see if you can bisect further?

Comment 6

[Robert Strong (they/them - no direct email)]

Mason, could you provide a link to the last installer that works and the first installer that doesn't?

Comment 7

[Mason Chang [Inactive] [:mchang]]

(In reply to Robert Strong [:rstrong] (use needinfo to contact me) from comment #6)
> Mason, could you provide a link to the last installer that works and the
> first installer that doesn't?

Last good - https://ftp.mozilla.org/pub/firefox/nightly/2016/04/2016-04-26-04-46-09-mozilla-central/firefox-49.0a1.en-US.win32.installer.exe

Last bad - https://ftp.mozilla.org/pub/firefox/nightly/2016/04/2016-04-27-03-02-15-mozilla-central/firefox-49.0a1.en-US.win32.installer.exe

Comment 8

[Robert Strong (they/them - no direct email)]

There don't appear to be any new files though there could be in omni.ja.

Could you try the following?

With 7-Zip extract both of those builds to separate directories.
Take the setup.exe from the extracted last good and replace the one in the extracted last bad.
Run the setup.exe in the last bad and install.
Check to see if it still crashes.

Note: the installer just copies the files it is given by the build system into the install directory and adds/modifies/deletes registry entries and shortcuts.

Comment 9

[Mason Chang [Inactive] [:mchang]]

(In reply to Robert Strong [:rstrong] (use needinfo to contact me) from comment #8)
> There don't appear to be any new files though there could be in omni.ja.
> 
> Could you try the following?
> 
> With 7-Zip extract both of those builds to separate directories.
> Take the setup.exe from the extracted last good and replace the one in the
> extracted last bad.
> Run the setup.exe in the last bad and install.
> Check to see if it still crashes.
> 
> Note: the installer just copies the files it is given by the build system
> into the install directory and adds/modifies/deletes registry entries and
> shortcuts.

I tried this and it is still crashing. What registry entries are being modified?

Comment 10

[Mason Chang [Inactive] [:mchang]]

(In reply to Ryan VanderMeulen [:RyanVM] from comment #5)
> Rob Strong or Stephen Pohl. That said, here's a regression range between
> that nightly and the previous one if you want to start looking for possible
> culprits:
> https://hg.mozilla.org/mozilla-central/
> pushloghtml?fromchange=9ce31e9f90cb&tochange=ab0044bfa1df858919797bcd6a9aef76
> a668cd4a
> 
> I can't say anything stands out there :\. Maybe try grabbing some of those
> builds from fx-team/inbound and see if you can bisect further?

Something in this changeset seems to have caused it with installer builds - https://treeherder.mozilla.org/#/jobs?repo=mozilla-central&revision=52072b6bec14

There are a couple of graphics bugs in there. Bug 1267253 and 1262187 look suspicious to me.

Comment 11

[Robert Strong (they/them - no direct email)]

If you tried this and it is still crashing then it isn't the installer since you used the known good installer to perform the installation.

The installer primarily sets up numerous shell integration registry entries for setting as default, protocol handlers, etc. which are used by the operating system, uninstall information, etc. the vast majority of which are not used by Firefox.

BTW: you should be able to get the same end result of a zip build just by extracting the installer and running the build from in the core directory. This might be due to the zip builds not containing the exact same files as the installer due to the way the files are packaged by the build system.

You should try running the extracted known bad from the core directory without installing to see if it also crashes.

Comment 12

[Mason Chang [Inactive] [:mchang]]

Bug 1267253 caused it, builds here - http://archive.mozilla.org/pub/firefox/tinderbox-builds/mozilla-inbound-win32/1461674276/

Also still reproducing it only when running it via the installer and not the downloaded .zip file.

Comment 13

[Mason Chang [Inactive] [:mchang]]

If I directly launch Firefox.exe from the install directory, I crash. If I directly launch firefox.exe from the extracted bad core directory, it works. If I copy the core directory over the install directory, it crashes. If I just copy the core directory to C:\Program Files\core\, it works. If I rename that core directory to Nightly, everything blows up.

Comment 14

[Mason Chang [Inactive] [:mchang]]

After much help from Robert, it basically seems like once we run Firefox.exe from C:\Program Files (x86)\Nightly, we get this crash. Installed or run from anywhere else, everything works fine.

George, I know you were working on some naming stuff, do you have any idea why this happens?

Comment 15

[Mason Chang [Inactive] [:mchang]]

The problem comes from using single buffering at [1]. It means we'll have a d3d11 compositor but a skia content client, and that'll cause us to use single buffering. This configuration is supposed to be supported and works EVERYWHERE other than when running from explicitly C:\Program Files (x86)\Nightly\firefox.exe. If I call the folder "Nightly" and move it to C:\, it works. Even moving it to C:\Program Files\Nightly\, everything works. So something, somewhere is basically doing something special at a hard coded path :/.


[1] https://dxr.mozilla.org/mozilla-central/source/gfx/layers/client/ContentClient.cpp?from=ContentClient.cpp#87

Comment 16

[Mason Chang [Inactive] [:mchang]]

Attachment: about:support

Comment 17

[Mason Chang [Inactive] [:mchang]]

Attachment: about:support w/ nvidia gpu

Doesn't appear to be GPU related as I can still reproduce with an nvidia quadro gpu instead of an intel HD GPU.

Comment 18

[Nicholas Nethercote [inactive]]

This is the #1 topcrash in Nightly 20160523030225, with 38 occurrences, which is a lot. All on Windows 7.

The earliest crash date I see is 2016-05-20. But we have build IDs going back to 20160427030215.

Comment 19

[Mason Chang [Inactive] [:mchang]]

(In reply to Nicholas Nethercote [:njn] from comment #18)
> This is the #1 topcrash in Nightly 20160523030225, with 38 occurrences,
> which is a lot. All on Windows 7.

That was probably me :)
 
> The earliest crash date I see is 2016-05-20. But we have build IDs going
> back to 20160427030215.

That was probably also me.