1610790 - GDATA exploit protection makes Firefox unusable after bug 1608645

Status: VERIFIED FIXED

(Core :: mozglue, defect, P1)

Description

[maggus.staab]

Attachment: firefox.png

User Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/81.0.4017.0 Safari/537.36 Edg/81.0.389.2

Steps to reproduce:

after the yesterdays update my firefox nightly started to no longer work.
starting firefox.exe opens the UI but does neither allow to navigate to any url nor does it allow to open internal pages like about:config or about:profile or similar

the firefox about dialog reports 74.0a1 (2020-01-21) (64-Bit) and no available updates.

I already tried to

• start with a fresh profile
• re-install over the existing instance
• uninstall + install a fresh firefox
• re-start the workstation
• re-download the installer and install again
• delete all folders named Mozilla, Mozilla Firefox or similar
• cleaned up the windows registry

related twitter conversation:
https://twitter.com/markusstaab/status/1219913504091668485

Actual results:

after starting firefox its not possible to navigate to any website. its neither possible via dropdown autocomplete chosen via mouse, nor by typing in a url.

its not even possible to stop firefox using the "X" windows icon.
exiting using the hamburger->exit works.

Expected results:

firefox should just work, allow open websites etc

Comment 1

[maggus.staab]

I tried installing firefox beta, firefox dev, firefox stable.. all work like expected. only firefox nightly does show the reported symptoms.

Comment 2

[BugBot [:suhaib / :marco/ :calixte]]

Bugbug thinks this bug should belong to this component, but please revert this change in case of error.

Comment 3

[maggus.staab]

at best someone could give me a pointer on how to clear everything firefox nightly might lock into this situation, so I could just re-install from scratch an re-import my old profile.

atm my productivity is hurt because all my browser history, bookmarks, etc are lost

Comment 4

[Pascal Chevrel:pascalc]

I am preemptively setting this to P1 Critical given that this looks like a major regression and I don't want that to slip into the cracks for out 74 release. If it turns out to be an isolated case and not a widespread issue, we will re-prioritize of course.

Comment 5

[Molly Howell (she/her) (mostly inactive)]

The bot moved this bug to an unhelpful component, so I'm putting it back.

Can I ask what's your reason for thinking bug 1610485 is related?

Comment 6

[[:philipp]]

do we need particular further information/diagnostics here to determine how we end up in a state where connectivity is lost?

Comment 7

[maggus.staab]

tbh. I am not 100% sure it is the cause.

I read about the problems with crashing firefox on twitter immediately after my nightly updated and got into this "corrupt" state.
dont take it as a verified information.

Comment 8

[Nhi Nguyen (:nhi)]

I don't think this NI was meant for me?

Comment 9

[[:philipp]]

sorry it was, as i was looking for someone from necko to chime in...

Comment 10

[Nhi Nguyen (:nhi)]

Ah, sorry, it wasn't clear when I first read it.

Honza, could you help with the question in comment #6? Thanks!

Comment 11

[Honza Bambas (:mayhemer)]

maggus, I'll ask for few questions, because this is definitely not a necko/networking issue. I assume you are on Windows.

• you said you tried to run firefox with a fresh profile, how exactly? (the more details the better)
• can you please copy-paste the content of "c:\Program Files\Firefox Nightly\platform.ini" file here?
• I'll ask you for a small test:
  • make sure there is not any Firefox Nightly process running (with Task Manager, shift-ctrl-esc)
  • create an empty directory like c:\temp\firefox-empty-profile
  • in the Firefox installation folder run firefox -profile c:\temp\firefox-empty-profile -no-remote
  • another variant is to run only firefox -profile c:\temp\firefox-empty-profile
    what happens then?
• when Firefox Nightly (with the empty profile) is running but not loading anything, can you run the Task Manager and look how many Firefox Nightly processes there is? the expected number is 6 or 7 child processes and one main process, all named Firefox Nightly

thanks.

Comment 12

[maggus.staab]

thx for getting back to me.

(In reply to Honza Bambas (:mayhemer) from comment #11)

maggus, I'll ask for few questions, because this is definitely not a necko/networking issue. I assume you are on Windows.

I am on windows10x64, latest updates installed

• you said you tried to run firefox with a fresh profile, how exactly? (the more details the better)

I had to use the ProfileManager to create a fresh one, because about:profiles did not open after starting the browser.
started with firefox.exe -P

• can you please copy-paste the content of "c:\Program Files\Firefox Nightly\platform.ini" file here?

[Build]
BuildID=20200121215203
Milestone=74.0a1
SourceRepository=https://hg.mozilla.org/mozilla-central
SourceStamp=79514ae28ef4b3508baf56fc8e89ef593a9b77b1

• I'll ask you for a small test:
  • make sure there is not any Firefox Nightly process running (with Task Manager, shift-ctrl-esc)
  • create an empty directory like c:\temp\firefox-empty-profile
  • in the Firefox installation folder run firefox -profile c:\temp\firefox-empty-profile -no-remote

nightly started and show the same symptomps as initially reported. the basic gui is visible, navigation does not work. most of the menu doesnt work. even the "X"-button (top right) of the window doesnt work.

• another variant is to run only firefox -profile c:\temp\firefox-empty-profile

same issue

what happens then?

• when Firefox Nightly (with the empty profile) is running but not loading anything, can you run the Task Manager and look how many Firefox Nightly processes there is? the expected number is 6 or 7 child processes and one main process, all named Firefox Nightly

I see 3, screenshot attached

thanks.

Comment 13

[maggus.staab]

Attachment: ff-nightly-processes.png

Comment 14

[maggus.staab]

fyi, the "dead guy" managed to download a update, and I am now on 74.0a1 (2020-01-22) (64-bit)
(the hamburger->about firefox dialog works and I copied this information from there)

still the same issues though.

Comment 16

[[:philipp]]

if it's possible to reproduce the issue in a new profile, perhaps you can also use run the tool from http://mozilla.github.io/mozregression/ in order to come up with an exact regression range? thank you

Comment 17

[maggus.staab]

ohh its getting interessting. the bug #1611077 which was closed as duplicate is reported by a co-worker of mine.

he experienced the same problem independently. we just realized we have the same error.

Comment 18

[maggus.staab]

ohh my.. just got a reported from a 3rd co-worker beeing affected.

as this is getting across several workstations, I assume it might e.g. related with our Antivirus or similar which might infere with the firefox update process

atm we are on gdata 14.2.1.6, screenshot attached

Comment 19

[maggus.staab]

Attachment: gdata.png

Comment 20

[Honza Bambas (:mayhemer)]

Thanks for all the information!

I suspected that something would block launching child processes which is now confirmed. If there are only 3, it means there is an antivirus that blocks us.

Only as a test (not a possible solution!!!) can you please create or rewrite c:\temp\firefox-empty-profile\prefs.js file with the following content:

user_pref("security.sandbox.content.level", 0);

and run firefox again as firefox -profile c:\temp\firefox-empty-profile ?

Other option is to use mozregression as :philipp mentioned in comment 16 to narrow down the exact change that started this. It should be very quick to find out, mozregression has a UI that is easy to use.

I want to narrow down the cause, if this is a firefox bug or an external software bug (in the gdata antivirus, likely)

thanks.

Comment 21

[maggus.staab]

(In reply to Honza Bambas (:mayhemer) from comment #20)

Thanks for all the information!

I suspected that something would block launching child processes which is now confirmed. If there are only 3, it means there is an antivirus that blocks us.

Only as a test (not a possible solution!!!) can you please create or rewrite c:\temp\firefox-empty-profile\prefs.js file with the following content:

user_pref("security.sandbox.content.level", 0);

and run firefox again as firefox -profile c:\temp\firefox-empty-profile ?

this makes it work again. I get a working browser and also 6-8 FF Nightly processes

Comment 22

[Honza Bambas (:mayhemer)]

Thanks, moving to sandboxing.

Comment 23

[maggus.staab]

seeing some strange error while bisecting (see attachment)

Comment 24

[maggus.staab]

Attachment: error-while-bisect.png

Comment 25

[maggus.staab]

Attachment: bisect-done.png

here we go

2020-01-23T17:21:25: INFO : Running autoland build built on 2020-01-21 01:03:39.395000, revision d2ac06fc
2020-01-23T17:21:25: DEBUG : using taskcluster root url https://firefox-ci-tc.services.mozilla.com
2020-01-23T17:21:25: DEBUG : using taskcluster route 'gecko.v2.autoland.shippable.revision.e0a14dad032f7327df45fcd119494f241d2476aa.firefox.win64-opt'
2020-01-23T17:21:27: DEBUG : using taskcluster root url https://firefox-ci-tc.services.mozilla.com
2020-01-23T17:21:27: DEBUG : using taskcluster route 'gecko.v2.autoland.shippable.revision.e29693ba69fca5d826e87114bc7f0ec52569d040.firefox.win64-opt'
2020-01-23T17:21:28: INFO : Launching c:\Users\mstaab\AppData\Local\Temp\tmpnyej_z\firefox\firefox.exe
2020-01-23T17:21:28: INFO : Application command: c:\Users\mstaab\AppData\Local\Temp\tmpnyej_z\firefox\firefox.exe --allow-downgrade  --wait-for-browser -profile c:\users\mstaab\appdata\local\temp\tmpm7moml.mozrunner
2020-01-23T17:21:28: INFO : application_buildid: 20200120224840
2020-01-23T17:21:28: INFO : application_changeset: d2ac06fcafcd2e32244a33cae326d1beac166f42
2020-01-23T17:21:28: INFO : application_display_name: Firefox Nightly
2020-01-23T17:21:28: INFO : application_id: {ec8030f7-c20a-464f-9b0e-13a3a9e97384}
2020-01-23T17:21:28: INFO : application_name: Firefox
2020-01-23T17:21:28: INFO : application_remotingname: firefox
2020-01-23T17:21:28: INFO : application_repository: https://hg.mozilla.org/integration/autoland
2020-01-23T17:21:28: INFO : application_vendor: Mozilla
2020-01-23T17:21:28: INFO : application_version: 74.0a1
2020-01-23T17:21:28: INFO : platform_buildid: 20200120224840
2020-01-23T17:21:28: INFO : platform_changeset: d2ac06fcafcd2e32244a33cae326d1beac166f42
2020-01-23T17:21:28: INFO : platform_repository: https://hg.mozilla.org/integration/autoland
2020-01-23T17:21:28: INFO : platform_version: 74.0a1
2020-01-23T17:21:39: INFO : Narrowed inbound regression window from [f88f5e88, f74adc43] (13 builds) to [d2ac06fc, f74adc43] (7 builds) (~2 steps left)
2020-01-23T17:21:48: INFO : Running autoland build built on 2020-01-21 02:53:25.996000, revision e0a14dad
2020-01-23T17:21:48: DEBUG : using taskcluster root url https://firefox-ci-tc.services.mozilla.com
2020-01-23T17:21:48: DEBUG : using taskcluster route 'gecko.v2.autoland.shippable.revision.5aa55ee754410ddaa8083044b3bb1adbc9743281.firefox.win64-opt'
2020-01-23T17:21:49: DEBUG : using taskcluster root url https://firefox-ci-tc.services.mozilla.com
2020-01-23T17:21:49: DEBUG : using taskcluster route 'gecko.v2.autoland.shippable.revision.4fe7a1c51fa56ba57ec55db4bfd2a46ab8f4a65a.firefox.win64-opt'
2020-01-23T17:21:51: INFO : Launching c:\Users\mstaab\AppData\Local\Temp\tmpaflfhn\firefox\firefox.exe
2020-01-23T17:21:51: INFO : Application command: c:\Users\mstaab\AppData\Local\Temp\tmpaflfhn\firefox\firefox.exe --allow-downgrade  --wait-for-browser -profile c:\users\mstaab\appdata\local\temp\tmparftyo.mozrunner
2020-01-23T17:21:51: INFO : application_buildid: 20200121002208
2020-01-23T17:21:51: INFO : application_changeset: e0a14dad032f7327df45fcd119494f241d2476aa
2020-01-23T17:21:51: INFO : application_display_name: Firefox Nightly
2020-01-23T17:21:51: INFO : application_id: {ec8030f7-c20a-464f-9b0e-13a3a9e97384}
2020-01-23T17:21:51: INFO : application_name: Firefox
2020-01-23T17:21:51: INFO : application_remotingname: firefox
2020-01-23T17:21:51: INFO : application_repository: https://hg.mozilla.org/integration/autoland
2020-01-23T17:21:51: INFO : application_vendor: Mozilla
2020-01-23T17:21:51: INFO : application_version: 74.0a1
2020-01-23T17:21:51: INFO : platform_buildid: 20200121002208
2020-01-23T17:21:51: INFO : platform_changeset: e0a14dad032f7327df45fcd119494f241d2476aa
2020-01-23T17:21:51: INFO : platform_repository: https://hg.mozilla.org/integration/autoland
2020-01-23T17:21:51: INFO : platform_version: 74.0a1
2020-01-23T17:22:02: INFO : Narrowed inbound regression window from [d2ac06fc, f74adc43] (7 builds) to [e0a14dad, f74adc43] (4 builds) (~2 steps left)
2020-01-23T17:22:02: INFO : Running autoland build built on 2020-01-21 02:55:09.531000, revision 5aa55ee7
2020-01-23T17:22:02: DEBUG : using taskcluster root url https://firefox-ci-tc.services.mozilla.com
2020-01-23T17:22:02: DEBUG : using taskcluster route 'gecko.v2.autoland.shippable.revision.e17821341a5252bace345e893cea18bb7eb51f69.firefox.win64-opt'
2020-01-23T17:22:04: INFO : Launching c:\Users\mstaab\AppData\Local\Temp\tmplfmb0l\firefox\firefox.exe
2020-01-23T17:22:04: INFO : Application command: c:\Users\mstaab\AppData\Local\Temp\tmplfmb0l\firefox\firefox.exe --allow-downgrade  --wait-for-browser -profile c:\users\mstaab\appdata\local\temp\tmpi_sihp.mozrunner
2020-01-23T17:22:04: INFO : application_buildid: 20200121002610
2020-01-23T17:22:04: INFO : application_changeset: 5aa55ee754410ddaa8083044b3bb1adbc9743281
2020-01-23T17:22:04: INFO : application_display_name: Firefox Nightly
2020-01-23T17:22:04: INFO : application_id: {ec8030f7-c20a-464f-9b0e-13a3a9e97384}
2020-01-23T17:22:04: INFO : application_name: Firefox
2020-01-23T17:22:04: INFO : application_remotingname: firefox
2020-01-23T17:22:04: INFO : application_repository: https://hg.mozilla.org/integration/autoland
2020-01-23T17:22:04: INFO : application_vendor: Mozilla
2020-01-23T17:22:04: INFO : application_version: 74.0a1
2020-01-23T17:22:04: INFO : platform_buildid: 20200121002610
2020-01-23T17:22:04: INFO : platform_changeset: 5aa55ee754410ddaa8083044b3bb1adbc9743281
2020-01-23T17:22:04: INFO : platform_repository: https://hg.mozilla.org/integration/autoland
2020-01-23T17:22:04: INFO : platform_version: 74.0a1
2020-01-23T17:22:20: INFO : Narrowed inbound regression window from [e0a14dad, f74adc43] (4 builds) to [5aa55ee7, f74adc43] (2 builds) (~1 steps left)
2020-01-23T17:22:20: DEBUG : Starting merge handling...
2020-01-23T17:22:20: DEBUG : Using url: https://hg.mozilla.org/integration/autoland/json-pushes?changeset=f74adc43b654c1e663d9e62be89effc5fb76d4bd&full=1
2020-01-23T17:22:21: DEBUG : Found commit message:
Bug 1608645 - Ensure FindExportAddressTableEntry can handle a modified Export Table.  r=aklotz

A third-party application can modify the export directory, the export address/name/ordinal
tables, or an entry in those tables.  If that happens, we will see an RVA is located outside
the mapped image and `RVAToPtr` returns null.  This patch makes sure we don't hit null AV
when modification is detected.

`FindExportAddressTableEntry` should not return a pointer to the modified table entry because
we dereference it in another process to cross-process detour.

Differential Revision: https://phabricator.services.mozilla.com/D59738

2020-01-23T17:22:21: DEBUG : Did not find a branch, checking all integration branches
2020-01-23T17:22:21: INFO : The bisection is done.
2020-01-23T17:22:21: INFO : Stopped

Comment 26

[Toshihito Kikuchi [:toshi]]

Ok, I confirmed this issue with G DATA Internet Security 2020 for Windows. Looks like they inject C:\Program Files (x86)\Common Files\G Data\AVKProxy\ExploitProtection64.dll into the browser process, which modifies ntdll's export table. With the fix for bug 1608645, detouring ntdll's functions fail when modification is detected, resulting in no content processes. Let's find out how we can be compatible with the variation of export table tampering such as 0Patch and G Data..

A temporal workaround would be to disable either the launcher process of Firefox or the "Exploit protections" feature of G Data.

Comment 27

[Ryan VanderMeulen [:RyanVM]]

Bug 1604008 and bug 1608645 have been backed out from Beta for 73.0b9 to avoid shipping this new problem to Beta users. Leaving the status set to affected for now, though, to make it clear that we'll eventually want to uplift this fix to avoid shipping the crashes from the regressing bugs.
https://hg.mozilla.org/releases/mozilla-beta/rev/404293e0e5aa

Comment 28

[maggus.staab]

any chance we could get a working FF nightly? atm the browser is not usable.

Comment 29

[Toshihito Kikuchi [:toshi]]

(In reply to maggus.staab from comment #28)

any chance we could get a working FF nightly? atm the browser is not usable.

Sorry for inconvenience. If it's not comfortable to disable the "Exploit protections" of G Data, the other option is to disable the launcher process of Firefox via Registry.

I think your system already has the following registry value. If you change its value to 0, the launcher process is disabled and this issue will not happen. The downside is without the launcher process, our dll blocklist does not work, meaning a third-party application may firefox unstable.

Key: HKCU\Software\Mozilla\Firefox\Launcher
Name: <Path to firefox.exe>|Browser
Type: REG_DWORD

Please note that the launcher process is automatically re-enabled after a new version of Nightly is installed.

Comment 30

[Thomas Siebert]

From G DATA's side, we sent out a signature update that should mitigate this problem for now. Also, we are having an in-depth look into that problem.

Comment 31

[Ryan VanderMeulen [:RyanVM]]

We're not planning to re-land the regressing bugs in Fx73 at this time, though we'll be monitoring the crash rates in case they spike after it goes to release.

Comment 32

[Thomas Siebert]

We were able to identify this problem.

Our Exploit Protection creates a modified EDT and EAT outside of the original image in order to monitor the access to the EAT. This seems to trigger some problems on your side. We looked at your code and assume that it should be enough if you just keep following the pointers/RVAs using RVAToPtrUnchecked instead of RVAToPtr in all places that want to access either the EDT or EAT.

And as a sidenote: It also looks like the compiled version of RVAToPtr only checks the upper limit of the image (prob. due to assuming the addtion of two unsigned values).

Comment 33

[Toshihito Kikuchi [:toshi]]

(In reply to Thomas Siebert from comment #32)

We were able to identify this problem.

Our Exploit Protection creates a modified EDT and EAT outside of the original image in order to monitor the access to the EAT. This seems to trigger some problems on your side. We looked at your code and assume that it should be enough if you just keep following the pointers/RVAs using RVAToPtrUnchecked instead of RVAToPtr in all places that want to access either the EDT or EAT.

And as a sidenote: It also looks like the compiled version of RVAToPtr only checks the upper limit of the image (prob. due to assuming the addtion of two unsigned values).

Thank you for investigation on your side. Unfortunately we cannot simply use RVAToPtrUnchecked. I think it would emerge Bug 1604008.

The problem is we still partially rely on the local process's export section data to detour a function in a remote process. I'm preparing a patch to implement GetProcAddress for a remote process without depending on any of the local data structure. In theory it would fix all of 1604008/1608645/1610790.

Comment 34

[Toshihito Kikuchi [:toshi]]

Attachment: Bug 1610790: Part 1 - Use wmain in mozglue/tests. r=handyman

This patch changes the entrypoint of test programs under mozglue/tests so that
a coming test program can handle a command string easily.

Comment 35

[Toshihito Kikuchi [:toshi]]

Attachment: Bug 1610790: Part 2 - Implement GetProcAddress for a remote process. r=handyman

This patch adds a function to get an exported function in a remote process.
We need this implementation to address Bug 1604008, Bug 1608645, and Bug 1610790.

When WindowsDllInterceptor detours a function in a remote process, we used the
native GetProcAddress locally, and then detours the returned address in the
target process. The problem is if the caller's export table was modified, the
address returned from GetProcAddress might be invalid in the target process,
which is Bug 1604008.

I implemented GetProcAddress depending on both local and remote process image,
but it caused two regressions Bug 1608645 and Bug 1610790 because multiple
applications modify firefox's export table in multiple ways, such as replacing
an entry of EAT, replacing an RVA to Export section, or etc.

With this patch, we can use PEExportSection<MMPolicy>::GetProcAddress to get
an exported function in a remote process without relying on any local data so
that it's not impacted by modification of the local export table.

Depends on D62314

Comment 36

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/95d33b309126
Part 1 - Use wmain in mozglue/tests.  r=handyman
https://hg.mozilla.org/integration/autoland/rev/8bc8a16881df
Part 2 - Implement GetProcAddress for a remote process.  r=handyman

Comment 37

[Ryan VanderMeulen [:RyanVM]]

Updating the flags for this bug to reflect the fact that all relevant work to fix this and the original regressing bugs has moved here.

Comment 38

[Cosmin Sabou [:CosminS]]

Push with failures: https://treeherder.mozilla.org/#/jobs?repo=autoland&resultStatus=testfailed%2Cbusted%2Cexception&revision=8bc8a16881dfc3455136e6e5a927fd57a2910469&selectedJob=288422626

Failure log: https://treeherder.mozilla.org/logviewer.html#?job_id=288422626&repo=autoland

Backout link: https://hg.mozilla.org/integration/autoland/rev/5eefe8c81ca536d27b11109e699975e0ffb94ffe

[task 2020-02-11T22:53:17.759Z] 22:53:17 INFO - clang-9: error: linker command failed with exit code 1 (use -v to see invocation)
[task 2020-02-11T22:53:17.760Z] 22:53:17 INFO - /builds/worker/workspace/build/src/config/rules.mk:520: recipe for target 'ShowSSEConfig' failed
[task 2020-02-11T22:53:17.760Z] 22:53:17 ERROR - make[4]: *** [ShowSSEConfig] Error 1
[task 2020-02-11T22:53:17.760Z] 22:53:17 INFO - make[4]: Leaving directory '/builds/worker/workspace/build/src/obj-firefox/mozglue/tests'
[task 2020-02-11T22:53:17.760Z] 22:53:17 INFO - make[4]: *** Waiting for unfinished jobs....
[task 2020-02-11T22:53:17.767Z] 22:53:17 INFO - make[4]: Entering directory '/builds/worker/workspace/build/src/obj-firefox/mozglue/tests'
[task 2020-02-11T22:53:17.767Z] 22:53:17 INFO - mozglue/tests/TestBaseProfiler
[task 2020-02-11T22:53:17.771Z] 22:53:17 INFO - /builds/worker/fetches/sccache/sccache /builds/worker/fetches/clang/bin/clang++ -std=gnu++17 -U_FORTIFY_SOURCE -D_FORTIFY_SOURCE=2 -fstack-protector-strong -Qunused-arguments -Qunused-arguments -Wall -Wbitfield-enum-conversion -Wempty-body -Wignored-qualifiers -Woverloaded-virtual -Wpointer-arith -Wshadow-field-in-constructor-modified -Wsign-compare -Wtype-limits -Wunreachable-code -Wunreachable-code-return -Wwrite-strings -Wno-invalid-offsetof -Wclass-varargs -Wempty-init-stmt -Wfloat-overflow-conversion -Wfloat-zero-conversion -Wloop-analysis -Wc++2a-compat -Wcomma -Wimplicit-fallthrough -Wunused-function -Wunused-variable -Werror=non-literal-null-conversion -Wstring-conversion -Wtautological-overlap-compare -Wtautological-unsigned-enum-zero-compare -Wtautological-unsigned-zero-compare -Wno-error=tautological-type-limit-compare -Wno-inline-new-delete -Wno-error=deprecated-declarations -Wno-error=array-bounds -Wno-error=backend-plugin -Wno-error=return-std-move -Wno-error=atomic-alignment -Wformat -Wformat-security -Wno-gnu-zero-variadic-macro-arguments -Wno-unknown-warning-option -D_GLIBCXX_USE_CXX11_ABI=0 -fno-sized-deallocation -fno-aligned-new -fcrash-diagnostics-dir=/builds/worker/artifacts -fno-exceptions -fno-strict-aliasing -fno-rtti -ffunction-sections -fdata-sections -fno-exceptions -fno-math-errno -pthread -pipe -g -Xclang -load -Xclang /builds/worker/workspace/build/src/obj-firefox/build/clang-plugin/libclang-plugin.so -Xclang -add-plugin -Xclang moz-check -O2 -fno-omit-frame-pointer -funwind-tables -Werror -o TestBaseProfiler /builds/worker/workspace/build/src/obj-firefox/mozglue/tests/TestBaseProfiler.list -lpthread -Wl,-z,noexecstack -Wl,-z,text -Wl,-z,relro -Wl,-z,nocopyreloc -Wl,-Bsymbolic-functions -Wl,--build-id=sha1 -fstack-protector-strong -rdynamic -Wl,-rpath-link,/builds/worker/workspace/build/src/obj-firefox/dist/bin -Wl,-rpath-link,/usr/local/lib -pie -ldl -lrt -Wl,--version-script,/builds/worker/workspace/build/src/build/unix/stdc++compat/hide_std.ld
[task 2020-02-11T22:53:17.773Z] 22:53:17 INFO - /builds/worker/fetches/binutils/bin/ld: /usr/lib/x86_64-linux-gnu/Scrt1.o: in function _start': [task 2020-02-11T22:53:17.773Z] 22:53:17 INFO - /build/eglibc-ZYONVs/eglibc-2.13/csu/../sysdeps/x86_64/elf/start.S:99: undefined reference tomain'
[task 2020-02-11T22:53:17.773Z] 22:53:17 INFO - clang-9: error: linker command failed with exit code 1 (use -v to see invocation)
[task 2020-02-11T22:53:17.773Z] 22:53:17 INFO - /builds/worker/workspace/build/src/config/rules.mk:520: recipe for target 'TestBaseProfiler' failed
[task 2020-02-11T22:53:17.773Z] 22:53:17 ERROR - make[4]: *** [TestBaseProfiler] Error 1
[task 2020-02-11T22:53:17.773Z] 22:53:17 INFO - make[4]: Leaving directory '/builds/worker/workspace/build/src/obj-firefox/mozglue/tests'
[task 2020-02-11T22:53:17.773Z] 22:53:17 INFO - /builds/worker/workspace/build/src/config/recurse.mk:74: recipe for target 'mozglue/tests/target' failed
[task 2020-02-11T22:53:17.774Z] 22:53:17 ERROR - make[3]: *** [mozglue/tests/target] Error 2
[task 2020-02-11T22:53:17.774Z] 22:53:17 INFO - make[3]: *** Waiting for unfinished jobs....

Comment 39

[Toshihito Kikuchi [:toshi]]

Updated D62314 (Part 1) to address the build error.

Comment 40

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/a6da36488887
Part 1 - Use wmain in mozglue/tests.  r=handyman
https://hg.mozilla.org/integration/autoland/rev/af694119d3c7
Part 2 - Implement GetProcAddress for a remote process.  r=handyman

Comment 41

[Cristina Coroiu [:ccoroiu]]

https://hg.mozilla.org/mozilla-central/rev/a6da36488887
https://hg.mozilla.org/mozilla-central/rev/af694119d3c7

Comment 42

[nemo]

I'm trying not to noise up the bug. But does "target milestone 75" mean that firefox will be non-functional on all the machines here until then, or is a backport to 73 still in the works? It was already a shock that the crashes that briefly appeared in nightly last month made it to stable.

Comment 43

[Ryan VanderMeulen [:RyanVM]]

See https://bugzilla.mozilla.org/show_bug.cgi?id=1604008#c54. Target Milestone just encapsulates when a fix hit mozilla-central.

Comment 44

[Toshihito Kikuchi [:toshi]]

(In reply to Thomas Siebert from comment #30)

From G DATA's side, we sent out a signature update that should mitigate this problem for now. Also, we are having an in-depth look into that problem.

Now that a new patch landed on Nightly, could you test G DATA's modification can coexist with the latest Nightly?

Comment 45

[Ryan VanderMeulen [:RyanVM]]

Here are links to test builds created on top of the current Beta and Release channels if people want to test this ahead of time. Any feedback would be greatly welcome, though the usual caveats apply about test builds, profiles, etc etc etc :-)

Beta74
Win32: https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/MDcm8btRQUOgDjEdqewfAQ/runs/0/artifacts/public/build/target.zip
Win64: https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/K4Bf9NZdQpqW4CvOFAkxbw/runs/0/artifacts/public/build/target.zip

Release73
Win32: https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/A59TwDaHTleuWI6znRS_iw/runs/0/artifacts/public/build/target.zip
Win64: https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/NpWeh5ZrSaa3ljIqNZKZ_Q/runs/0/artifacts/public/build/target.zip

Comment 46

[nemo]

Whatever the heck was crashing over here which was apparently neither 0patch nor gdata, is fixed over here using the 64 bit 73 windows build \o/

Comment 47

[Cristian Fogel [:cfogel]]

As request for PI-486 check, results for the test session can be found in this document.

Comment 48

[Toshihito Kikuchi [:toshi]]

Comment on attachment 9125595 [details]
Bug 1610790: Part 2 - Implement GetProcAddress for a remote process. r=handyman

Beta/Release Uplift Approval Request

• User impact if declined: This is a startup crash if 0Patch (and possibly some other antivirus apps) is installed in the system. This has been one of the top crashes since 73 Beta.
• Is this code covered by automated tests?: Yes
• Has the fix been verified in Nightly?: Yes
• Needs manual test from QE?: No
• If yes, steps to reproduce:
• List of other uplifts needed: None
• Risk to taking this patch: Medium
• Why is the change risky/not risky? (and alternatives if risky): Given the complexity of the change and we already shipped two regressions, the risk is medium. The latest fix is to implement the remote version of GetProcAddress. If there is something bad in the new logic, we'll have a new crash. To reduce the risk, this patch contains a thorough unit-test. QA team has already verified the patch on Beta and Release and no regression was observed.
• String changes made/needed:

Comment 49

[Ryan VanderMeulen [:RyanVM]]

Comment on attachment 9125593 [details]
Bug 1610790: Part 1 - Use wmain in mozglue/tests. r=handyman

Approved for 74.0b3 so we can get more feedback.

Comment 50

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/releases/mozilla-beta/rev/22b7f25548bb
https://hg.mozilla.org/releases/mozilla-beta/rev/a0b47c03da96

Comment 52

[Myron]

Confirmation that FF beta 74.0b3 (64-bit) and 0patch v19.11.15.10650 appear to be operating correctly with 0patch's mitigation removed.

Comment 53

[Cristian Fogel [:cfogel]]

Verified with 75.0a1 (2020-02-16) on Windows 10 and 7.

Comment 54

[Ryan VanderMeulen [:RyanVM]]

Comment on attachment 9125593 [details]
Bug 1610790: Part 1 - Use wmain in mozglue/tests. r=handyman

Fixes a frequent crash seen by users in the wild after 73.0 was released. Approved for 73.0.1.

Comment 55

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/releases/mozilla-release/rev/47300b0bbde2
https://hg.mozilla.org/releases/mozilla-release/rev/2798e8ae6c25

Comment 56

[Thomas Siebert]

(In reply to Toshihito Kikuchi [:toshi] from comment #44)

(In reply to Thomas Siebert from comment #30)

From G DATA's side, we sent out a signature update that should mitigate this problem for now. Also, we are having an in-depth look into that problem.

Now that a new patch landed on Nightly, could you test G DATA's modification can coexist with the latest Nightly?

I can confirm your patch fixes the problem, and can coexist with our mitigation.

Comment 57

[Toshihito Kikuchi [:toshi]]

(In reply to Thomas Siebert from comment #56)

I can confirm your patch fixes the problem, and can coexist with our mitigation.

That's great news. Thank you for confirming it!

Comment 58

[Ryan VanderMeulen [:RyanVM]]

Added to the Firefox 73.0.1 release notes:

Fixed crashes on Windows systems running third-party security software such as 0patch or G DATA

Comment 59

[Cristian Fogel [:cfogel]]

Verified as well with 73.0.1 on Windows 10/7(x64).

Comment 60

[nemo]

Firefox 73.0.1 confirmed working over here too with our (non-0patch/GData - Cylance?) setup.. \o/

Comment 61

[Firefox Bug Husbandry Bot]

Please specify a root cause for this bug. See :tmaity for more information.