1629361 - Firefox 75 no longer able to load content with Netsupport School present

Status: VERIFIED FIXED

(Firefox :: Launcher Process, defect)

Description

[[:philipp]]

After the release of Firefox 75 we got a number of reports that Firefox is no longer functional (not able to load content or internal pages and not properly closing), similar to the symptoms of bug 1610790 and bug 1614885:
https://support.mozilla.org/en-US/questions/firefox?owner=all&tagged=bug1629361&show=all

A user has linked this to the presence of the external Netsupport School software on an affected system. I could reproduce that by installing a trial version of the tool, afterwards Firefox turned unfuctional - this problem regressed with the changes from bug 1615308.

Comment 1

[Toshihito Kikuchi [:toshi]]

I got a local repro. It seems that NSCommonHook64.dll was injected via SetWindowHookEx, which modified an offset to NtOpenFile in firefox.exe's import table for ntdll.dll, thus launching a content process from the browser process failed. There is no perfect solution for this. To disable the launcher process will be a mitigation.

Comment 2

[Toshihito Kikuchi [:toshi]]

What makes the case worse is we don't automatically disable the launcher process after this happens. As a mitigation, we can disable the launcher process when the browser process fails to start a sandbox process.

One possible fix would be to cache IAT in the browser process before injection via window hook happens, and copy that cache into a sandbox process.

Comment 3

[Toshihito Kikuchi [:toshi]]

Attachment: Bug 1629361 - Disable the launcher process when a content process fails to start. r=mhowell

If a third-party application modifies IAT of ntdll.dll in the browser process
after process launch, the browser process fails to launch a sandbox process,
resulting in a situation where a window is opened without any functionality.

This patch is to mitigate that situation by disabling the launcher process
when the browser process fails to launch a sandbox process.

Comment 4

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/b808c96379c4
Disable the launcher process when a content process fails to start.  r=mhowell

Comment 5

[Cristina Coroiu [:ccoroiu]]

https://hg.mozilla.org/mozilla-central/rev/b808c96379c4

Comment 6

[[:philipp]]

thank you for the patch - i can confirm that this fixed the issue for me on nightly as well.

could you request uplift to beta, since the problem has some major impact for affected users (and there are a few reports from users on 75 who don't have netsupport so there may be more software out there triggering this condition)?

Comment 7

[Toshihito Kikuchi [:toshi]]

(In reply to [:philipp] from comment #6)

thank you for the patch - i can confirm that this fixed the issue for me on nightly as well.

could you request uplift to beta, since the problem has some major impact for affected users (and there are a few reports from users on 75 who don't have netsupport so there may be more software out there triggering this condition)?

That's a great news! Thank you for confirming the behavior. It's expected that there are other applications causing the same problem and this patch mitigates it, too. I agree to uplift this to beta.

Comment 8

[Toshihito Kikuchi [:toshi]]

Comment on attachment 9140448 [details]
Bug 1629361 - Disable the launcher process when a content process fails to start. r=mhowell

Beta/Release Uplift Approval Request

• User impact if declined: If NetSupport School is installed, Firefox is launched without any functionality like bug 1614885 because all content processes failed to start. The situation persists, meaning next time a user starts Firefox, she will hit the same situation unless she modifies the registry manually.
• Is this code covered by automated tests?: No
• Has the fix been verified in Nightly?: Yes
• Needs manual test from QE?: No
• If yes, steps to reproduce:
• List of other uplifts needed: None
• Risk to taking this patch: Low
• Why is the change risky/not risky? (and alternatives if risky): The patch is not a perfect solution but a straightforward mitigation. If the browser process fails to start a sandbox process for some reason, this patch automatically disables the launcher process by updating the registry value so that a user will see this blank-page situation only once.
• String changes made/needed: None

Comment 9

[Ryan VanderMeulen [:RyanVM]]

Comment on attachment 9140448 [details]
Bug 1629361 - Disable the launcher process when a content process fails to start. r=mhowell

Definitely not ideal, but I agree that this is better than the alternative at the moment. Approved for 76.0b5.

Comment 10

[Ryan VanderMeulen [:RyanVM]]

BTW, is this something we should be recording Telemetry for?

Comment 11

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/releases/mozilla-beta/rev/675435f7739e

Comment 12

[Toshihito Kikuchi [:toshi]]

(In reply to Ryan VanderMeulen [:RyanVM] from comment #10)

BTW, is this something we should be recording Telemetry for?

Good point. Ideally we want to collect failures of starting a sandbox process as the launcher process failure ping. That data would indicate how often third-party applications inject code into our processes asymmetrically i.e. injecting into only the browser process without touching the launcher process intentionally or unintentionally. Let's file a tracking bug.

Comment 13

[[:philipp]]

confirmation from affected users that the fix is working as intended on beta:

• (netsupport user) https://support.mozilla.org/en-US/questions/1283949#answer-1306100
• (non-netsupport user, root cause not identified) https://old.reddit.com/r/firefox/comments/g0bwuk/firefox_crashes_and_now_cannot_display_anything/fn9cb6p/

Comment 14

[Roland Tanglao needinfo please :rolandtanglao, :mohnkuchen, :adobo, :sinigang, :roland]

1 more SUMO confirmation that this fixes other software in this case: crosstec schoolvue instead of netsupport school:

• upgrade from 74 to 75 the browser won't go to any site. The lovely new big address bar shows up, but that's about it. Other browsers no problem

Comment 15

[Daniel Bodea [:danibodea]]

Setting general status based on the verification of firefox75 and firefox76. Please revert if incorrect. In theory, firefox75 still affected.