1614885 - Firefox 73 no longer able to load content when launched in Win7 compatibility mode

Status: VERIFIED FIXED

(Core :: mozglue, defect, P1)

Description

[[:philipp]]

[Tracking Requested - why for this release]:
In the first few hours of the 73 rollout there are numerous user reports that Firefox lost the ability to open any urls (same symptoms as bug 1610790) - this is sometimes accompanied by the application error code 0xc0000005 in the windows event log for firefox.

i'm able to somewhat reproduce that on windows 10 by setting the compatibility settings for firefox to run as "windows 7", though it's not possible to tell if this same configuration was in place for everyone affected that reported these symptoms.

Comment 1

[[:philipp]]

https://support.mozilla.org/en-US/questions/firefox?owner=all&tagged=bug1614885&show=all
https://old.reddit.com/r/firefox/comments/f2d5cu/nothing_will_load/
https://old.reddit.com/r/firefox/comments/f2s6og/firefox_doesnt_open_any_pages/

Comment 2

[Ryan VanderMeulen [:RyanVM]]

Given the reports we're getting, this sounds bad. Also, philipp confirmed that bug 1610790 does not fix this on Nightly builds.

Comment 3

[Toshihito Kikuchi [:toshi]]

I also reproduced the issue by setting the compatibility mode of firefox.exe to Win7. The root cause of this problem is the mismatch of Import Table of firefox.exe between the browser process and the content process, while the root cause of bug 1610790 is the mismatch of Export Table. We need a different fix.

Comment 4

[[:philipp]]

firefox 73.0a1 build 20191207092617 was the first nightly where the problem reproduces turning on win7 compat mode - this is the list of bugs with a patch landing in this build: https://bugzilla.mozilla.org/buglist.cgi?bug_id=1511958%2C1601527%2C1601963%2C1571285%2C1596918%2C1592427%2C1602089%2C1601700%2C1600974%2C1601103%2C1601508%2C1522830%2C1601851%2C1602105%2C1601813%2C1599444%2C1601756%2C1598212%2C1548274%2C1593841%2C1598377%2C1601397%2C1601622%2C1565332%2C1581119%2C&list_id=15106064

so perhaps bug 1522830 is a potential regressor.

Comment 5

[Ryan VanderMeulen [:RyanVM]]

That was the regressor for bug 1604008, so that would make sense.

Comment 6

[Toshihito Kikuchi [:toshi]]

When the compat mode is on, some ntdll functions in firefox's IAT are replaced with AcLayers.dll's functions.

2.6 RtlAllocateHeap@712 00007ff6`4e980710 00007ff9`f5806c20 AcLayers!NS_FaultTolerantHeap::APIHook_RtlAllocateHeap
2.13 RtlFreeHeap@985 00007ff6`4e980748 00007ff9`f5806e90 AcLayers!NS_FaultTolerantHeap::APIHook_RtlFreeHeap
2.15 RtlGetVersion@1071 00007ff6`4e980758 00007ff9`f5822190 AcLayers!NS_Win7RTMVersionLie::APIHook_RtlGetVersion
2.19 RtlReAllocateHeap@1315 00007ff6`4e980778 00007ff9`f5807360 AcLayers!NS_FaultTolerantHeap::APIHook_RtlReAllocateHeap

However, this replacement happens after we copied IAT entries into a child process. Thus there is a time window where AcLayers's address is set but the dll is not loaded. This causes 0xc0000005 when the process tries to call one of AcLayers functions.

00 0000009b`ba7ff050 00007ffa`04af99dc apphelp!SepIatPatch+0xc1
01 0000009b`ba7ff0e0 00007ffa`04aebb8a apphelp!SepRouterHookImportedApi+0xdc2c
02 0000009b`ba7ff1c0 00007ffa`04aeb808 apphelp!SepRouterHookIAT+0x2da
03 0000009b`ba7ff260 00007ffa`0996caef apphelp!SE_DllLoaded+0xa8
04 0000009b`ba7ff2c0 00007ffa`0996c98f ntdll!LdrpSendPostSnapNotifications+0x12b
05 0000009b`ba7ff330 00007ffa`0996aa4b ntdll!LdrpNotifyLoadOfGraph+0x4f
06 0000009b`ba7ff360 00007ffa`09a14546 ntdll!LdrpPrepareModuleForExecution+0x73
07 0000009b`ba7ff3a0 00007ffa`09a01df5 ntdll!LdrpInitializeProcess+0x1dbe
08 0000009b`ba7ff7e0 00007ffa`099b1853 ntdll!_LdrpInitialize+0x50589
09 0000009b`ba7ff880 00007ffa`099b17fe ntdll!LdrpInitialize+0x3b
0a 0000009b`ba7ff8b0 00000000`00000000 ntdll!LdrInitializeThunk+0xe

I have not yet come up with a nice solution, but as a mitigation, I think we should stop trying to enable the blocklist in a child process if the launcher process fails to enable it in the browser process. Then this issue will become a one-time issue.

Comment 7

[Toshihito Kikuchi [:toshi]]

Attachment: Bug 1614885 - Do not attempt to bootstrap a child process if the launcher failed to boostrap the browser process. r=aklotz

Bug 1522830 added the call to InitializeDllBlocklistOOP in SandboxBroker::LaunchApp
to enable the new dll blocklist and telemetry in sandbox processes. If the browser
process fails to bootstrap a process for some reason, firefox starts without any crash
nor any content processes because of that change.

What is worse is that this problem persists even after the launcher process was disabled.
To mitigate it, this patch stops an attempt to bootstrap a child process if the launcher
process already failed to do it. With this, if something bad happens in the first launch,
the launcher process is automatically disabled via registry and next time firefox will work
normally. So a user will see the launching problem only once.

We will follow up the bootstrap issue.

Comment 8

[Toshihito Kikuchi [:toshi]]

In https://old.reddit.com/r/firefox/comments/f2d5cu/nothing_will_load/, one user mentions Exploit Protection of Windows Defender.

In Windows 10, I had added Firefox to the "Exploit Protection" with a whole bunch of rules enabled.
It seems as of Firefox 73, the following 2 rules must be disabled:

• Validate stack integrity (StackPivot)
• Import address filtering (IAF)

This has been fixed in Nightly as bug 1592486. If we hit this issue in Beta or Release, the 0xc0000142 popup (instead of 0xc0000005) will be appeared once, and then hitting a no-content-process situation next time. The patch above mitigates this Windows Defender issue as well as the Windows compat mode issue.

Comment 9

[Daniel Bodea [:danibodea]]

Attachment: at launch.png

I have reproduced this issue In Nightly v75.0a1 from 2020-02-12 and I thought some screenshots might help to not confuse issues between them.

This is what I see when I open Nightly with a newly created profile on Windows 10, but Nightly is set to be opened in compatibility mode for Windows 7.

Loading any link results in infinite loading, refresh button is disabled, the user can switch tabs.

Comment 10

[Daniel Bodea [:danibodea]]

Attachment: after clicking on the second tab.png

This is what the other tab displays.
Attempting to load any other link in the address bar will result in infinite loading, the "refresh" button is disabled, the browser is basically unusable.

Comment 11

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/96ae269611a7
Do not attempt to bootstrap a child process if the launcher failed to boostrap the browser process.  r=aklotz

Comment 12

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/mozilla-central/rev/b0b5ea1916c5
Do not attempt to bootstrap a child process if the launcher failed to boostrap the browser process.  r=aklotz

Comment 13

[Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)]

New Windows Nightlies have been requested.

Comment 14

[Cristian Fogel [:cfogel]]

As request for PI-486 check, results for the test session can be found in this document.

Comment 15

[Toshihito Kikuchi [:toshi]]

I opened bug 1615308 to follow up the 0xc0000005 error popup.

Comment 16

[Cristian Brindusan [:cbrindusan]]

https://hg.mozilla.org/mozilla-central/rev/96ae269611a7

Comment 17

[Toshihito Kikuchi [:toshi]]

Opened bug 1615401 for the 32bit-specific issue. It's not a regression. I saw this 32bit crash on 72.0.2, too.

Comment 18

[Toshihito Kikuchi [:toshi]]

Comment on attachment 9126197 [details]
Bug 1614885 - Do not attempt to bootstrap a child process if the launcher failed to boostrap the browser process. r=aklotz

Beta/Release Uplift Approval Request

• User impact if declined: Firefox launches with no content processes, meaning no functionality, if Windows compatibility mode for firefox.exe is set to Win7. This problem persists and there is no chance to run automatic update.
• Is this code covered by automated tests?: No
• Has the fix been verified in Nightly?: Yes
• Needs manual test from QE?: No
• If yes, steps to reproduce:
• List of other uplifts needed: None
• Risk to taking this patch: Low
• Why is the change risky/not risky? (and alternatives if risky): The change is small enough to understand the impact. We skip to bootstrap a child process if we failed already. No bootstrap in a content process is the old behavior before the regressing change bug 1522830, so this is a fallback to the original behavior. Manual testing by QA team is done and no new regression was observed.
• String changes made/needed: None

Comment 20

[Ryan VanderMeulen [:RyanVM]]

Comment on attachment 9126197 [details]
Bug 1614885 - Do not attempt to bootstrap a child process if the launcher failed to boostrap the browser process. r=aklotz

Approved for 74.0b3 so we can get wider feedback.

Comment 21

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/releases/mozilla-beta/rev/df6323bac412

Comment 24

[Cristian Fogel [:cfogel]]

Verified with 75.0a1 (2020-02-16) on Windows 10 and 7.

Comment 25

[Ryan VanderMeulen [:RyanVM]]

Comment on attachment 9126197 [details]
Bug 1614885 - Do not attempt to bootstrap a child process if the launcher failed to boostrap the browser process. r=aklotz

Works around a non-functional browser for users in some circumstances since updating to 73.0. Approved for 73.0.1.

Comment 26

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/releases/mozilla-release/rev/806d21c68f12

Comment 27

[Ryan VanderMeulen [:RyanVM]]

Added to the Firefox 73.0.1 release notes:

Fixed loss of browser functionality in certain circumstances such as running in Windows compatibility mode or having custom anti-exploit settings

Comment 28

[Cristian Fogel [:cfogel]]

Verified with 73.0.1 on Windows 10/7(x64).

Comment 32

[Firefox Bug Husbandry Bot]

Please specify a root cause for this bug. See :tmaity for more information.