1525588 - 2019-02-06 ARM64 Windows build won't start

Status: VERIFIED FIXED

(Core :: General, defect)

Description

[Andrew Overholt [:overholt]]

I can reliably reproduce with either an installation of the latest nightly (today's) or upgrading to it.

STR

• download the Windows ARM64 installer from http://ftp.mozilla.org/pub/firefox/nightly/2019/02/2019-02-04-21-42-59-mozilla-central/
• check for updates and update to the 2019-02-06 build
• restart

Expected

• just works.

Actual

• I see 2 processes start up in Task Manager but never get my main window.

nbp (CCd) reported the same experience. I think we're both on Lenovo Yoga C630 devices.

Comment 1

[Ryan VanderMeulen [:RyanVM]]

Wonder if it's related to bug 1485016.

Comment 2

[Andrew Overholt [:overholt]]

FWIW I can start the 2019-02-06 build on x86_64 Windows 10.

Comment 3

[Tom Ritter [:tjr]]

I expect this was me and I will cook up a patch and confirm with overholt

Comment 4

[(Away)]

We're hitting ntdll!LdrpHandleInvalidUserCallTarget on:

xul!std::sys::windows::thread_local::on_tls_callback [/rustc/9fda7c2237db910e41d6a712e9a2139b352e558b/src\libstd\sys\windows\thread_local.rs @ 211]:

This happens during ntdll!LdrpCallTlsInitializers within the loading of libxul.

Maybe we don't have a CFG story for rust code? I wonder why this isn't a problem on x86_64.

Comment 5

[(Away)]

We do reach on_tls_callback on x86_64, without any CFG complains along the way.

Looking at dumpbin -loadconfig, an x86_64 xul.dll has:

0000000185735000 Guard CF address of check-function pointer
0000000185735008 Guard CF address of dispatch-function pointer
0000000185384B8D Guard CF function table

But on aarch64, xul.dll is missing the dispatch function:

0000000186485000 Guard CF address of check-function pointer
0000000000000000 Guard CF address of dispatch-function pointer
000000018606D98D Guard CF function table

Comment 6

[(Away)]

So the dispatch-function thing may be a red herring; Microsoft's own DLLs don't have one either.

Comment 7

[(Away)]

I'll continue to look at it but at this point it's not looking like I'm going to come up with a quick fix. We should probably disable CFG on arm64 for the time being. Maybe even land directly on m-c.

Comment 8

[(Away)]

OK, so I found that xul!std::sys::windows::thread_local::on_tls_callback is on the approved targets list on x86_64 (it doesn't resolve by name so I had to look it up by address) but not on aarch64.

Comment 9

[Tom Ritter [:tjr]]

Attachment: Bug 1525588 - Do not enable CFG on ARM builds, as it causes undiagnosed failures r?dmajor

Comment 10

[(Away)]

Comment on attachment 9041850 [details] [diff] [review]
Bug 1525588 - Do not enable CFG on ARM builds, as it causes undiagnosed failures r?dmajor

Let's use target.cpu instead of CONFIG.

Comment 11

[Tom Ritter [:tjr]]

Attachment: Bug 1525588 - Do not enable CFG on ARM builds, as it causes undiagnosed failures r?dmajor

Comment 12

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/ea669e7b7e93
Do not enable CFG on ARM builds, as it causes undiagnosed failures. r=dmajor

Comment 13

[Joel Byford]

Sorry, didn't see this is duplicated in bug 1525806. There's a WER file attached there, but let me know if you need more.

Comment 15

[Razvan Maries]

https://hg.mozilla.org/mozilla-central/rev/ea669e7b7e93

Comment 19

[Tom Ritter [:tjr]]

Comment on attachment 9041858 [details] [diff] [review]
Bug 1525588 - Do not enable CFG on ARM builds, as it causes undiagnosed failures r?dmajor

Beta/Release Uplift Approval Request

Feature/Bug causing the regression

None

User impact if declined

Necessary if Bug 1485016 is uplifted

Is this code covered by automated tests?

Yes

Has the fix been verified in Nightly?

Yes

Needs manual test from QE?

No

If yes, steps to reproduce

List of other uplifts needed

None

Risk to taking this patch

Low

Why is the change risky/not risky? (and alternatives if risky)

It's disabling the feature for ARM builds.

String changes made/needed

Comment 20

[Liz Henry (:lizzard) (relman/hg->git project)]

Comment on attachment 9041858 [details] [diff] [review]
Bug 1525588 - Do not enable CFG on ARM builds, as it causes undiagnosed failures r?dmajor

OK for beta uplift - please land the patch in bug 1523003 first, then the patch in bug 1485016, then this patch in bug 1525588.

Comment 21

[Natalia Csoregi [:nataliaCs]]

https://hg.mozilla.org/releases/mozilla-beta/rev/65897ef66dab

Comment 22

[Daniel Bodea [:danibodea]]

I can confirm the verification of the issue in firefox67 since the Aarch64 build starts and runs properly on Lenovo Yoga C630-13Q50 with Windows 10 Home (v1803). Verified using: Nightly v67.0a1 (2019-02-11) (64-bit) (Aarch64 build)
There isn't any build for beta yet, so I will leave the general status as before.
Thank you.

Comment 23

[Rob Thijssen [:grenade (EET/UTC+0300)]]

problem was back for me today. i ran the nightly updater but firefox failed to restart.
to fix:

• i downloaded latest aarch64 nightly installer and ran a fresh install
• i ran "c:\Program Files\Firefox Nightly\firefox.exe" -p and deleted my one and only profile, then created a new profile, firefox still wouldn't run
• i found a bunch of firefox processes running in task manager and killed them all
• i deleted the contents of %APPDATA%\Mozilla\Firefox\Profiles (there were two profile folders present)
• i created a new default profile

firefox started after this.

Comment 24

[(Away)]

If the browser starts at all, even if it takes considerable gymnastics, then it would have to be a different issue from the one tackled here: the nature of the CFG bug would have prevented you from being able to launch ever. It would be better to track the new issue as a separate bug. (FWIW I couldn't reproduce it just now)

Comment 25

[Daniel Bodea [:danibodea]]

Rob, can you please log a different issue for this? please specify the exact build you used. Thanks.

Comment 26

[Rob Thijssen [:grenade (EET/UTC+0300)]]

(In reply to Bodea Daniel [:danibodea] from comment #25)

Rob, can you please log a different issue for this? please specify the exact build you used. Thanks.

i suspect that if my issue was not related to this bug, then it was just a quirk of my system setup. the fact that there were running firefox processes in the background might have prevented the profile manager from completely deleting a running profile and may have been what prevented firefox from starting after the clean install.

i initially commented above as a result of googling for a solution for firefox not restarting when i hit the update button and arriving at this and other aarch64 specific bugs. i don't think i have a new issue to report. my comment was just to help people who arrived here looking for a way to get ff working again if they happen to be in the same situation of having some running ff processes in the background.

apologies for causing confusion on this bug.