1630860 - Scroll wheel appears to no longer scroll pages (77.0a1 (2020-04-16), windows)

Status: RESOLVED WORKSFORME

(Core :: Security: Process Sandboxing, defect, P1)

Description

[Davyd McColl]

User Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:77.0) Gecko/20100101 Firefox/77.0

Steps to reproduce:

Regular nightly updates -- repros in this version, but also the immediately proceeding one (when it was happening, I allowed Nightly to update, thinking that perhaps it was already fixed)

Actual results:

Scroll wheel on pages doesn't scroll the page. I can scroll other applications and ctrl+scrollwheel changes the zoom level, as expected, so the mouse is working and Firefox can see events.

Expected results:

Page should scroll

Comment 1

[Davyd McColl]

Note, this applies to all areas of FIrefox -- I can't scroll the Options page either. Private browser -- same result. So I don't think it's an extension interfering here.

Comment 2

[Giovanni [:Gioxx] (Mozilla Italia)]

Same problem here.
Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:77.0) Gecko/20100101 Firefox/77.0

Comment 3

[Francesco Lodolo [:flod]]

Got a similar report from a friend.

Comment 4

[:Gijs (out for now; he/him)]

This seems bad, but several of us cannot reproduce it on our end. Can you post the raw data from about:support (help > troubleshooting information, then click 'copy raw data to clipboard', then paste in the attachment box at https://bugzilla.mozilla.org/attachment.cgi?bugid=1630860&action=enter ) ?

Comment 5

[:Gijs (out for now; he/him)]

Rares, is QE seeing this on any of your machines?

Comment 6

[:Gijs (out for now; he/him)]

(In reply to Giovanni [:Gioxx] (Mozilla Italia) from comment #2)

Same problem here.
Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:77.0) Gecko/20100101 Firefox/77.0

Same question for you as comment #4 . :-)

Comment 7

[Giovanni [:Gioxx] (Mozilla Italia)]

Attachment: about:support from Gioxx

Comment 8

[Giovanni [:Gioxx] (Mozilla Italia)]

(In reply to :Gijs (he/him) from comment #6)

(In reply to Giovanni [:Gioxx] (Mozilla Italia) from comment #2)

Same problem here.
Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:77.0) Gecko/20100101 Firefox/77.0

Same question for you as comment #4 . :-)

Done.

Comment 9

[Pascal Chevrel:pascalc]

I can't reproduce but I am on Linux with a touchpad not an external mouse. I am guessing that Gijs and Flod are on macOS so maybe this is Windows only or it only happens with a mouse or specific mice.

Comment 10

[:Gijs (out for now; he/him)]

(In reply to Pascal Chevrel:pascalc from comment #9)

I am guessing that Gijs and Flod are on macOS so maybe this is Windows only or it only happens with a mouse or specific mice.

I and Masayuki tested on Windows. It could be related to a specific mouse but then it's surprising that so many people started seeing it.

(In reply to Giovanni [:Gioxx] (Mozilla Italia) from comment #8)

Done.

Thanks. Would you mind checking if you see the same thing in a new, clean profile (keeping the regular one; https://support.mozilla.org/kb/profile-manager-create-and-remove-firefox-profiles )? And, just to confirm, your machine doesn't have a touch screen, or does it?

Comment 11

[:Gijs (out for now; he/him)]

It's (Orthodox) Easter, redirecting from Rares to Tania for comment #5

Comment 12

[Cameron McCormack (:heycam)]

For those of you who can reproduce, are you using a mouse or a trackpad to scroll?

Comment 13

[Cameron McCormack (:heycam)]

(I guess "wheel" in comment 0 means scrolling with a mouse wheel though. I wouldn't normally expect mouse wheel scrolling to be affected by weird window class/name issues that trackpad drivers can be.)

Comment 14

[Cameron McCormack (:heycam)]

For those of you who can reproduce, could you try using mozregression to narrow down a particular change that introduced this problem?

https://mozilla.github.io/mozregression/quickstart.html

Comment 15

[Giovanni [:Gioxx] (Mozilla Italia)]

Attachment: Clean profile, same problem

Comment 16

[Giovanni [:Gioxx] (Mozilla Italia)]

Attachment: mozregression log

Comment 17

[Giovanni [:Gioxx] (Mozilla Italia)]

(In reply to :Gijs (he/him) from comment #10)

Thanks. Would you mind checking if you see the same thing in a new, clean profile (keeping the regular one; https://support.mozilla.org/kb/profile-manager-create-and-remove-firefox-profiles )? And, just to confirm, your machine doesn't have a touch screen, or does it?

Clean profile created and started, same problem.
You can find about:support information here: https://bugzilla.mozilla.org/attachment.cgi?id=9141190
My laptop doesn't have touch screen.

(In reply to Cameron McCormack (:heycam) from comment #12)

For those of you who can reproduce, are you using a mouse or a trackpad to scroll?

External Keyboard + Mouse kit (Lenovo, but I can reproduce the same problem with another Logitech kit) connected with dongle.

(In reply to Cameron McCormack (:heycam) from comment #14)

For those of you who can reproduce, could you try using mozregression to narrow down a particular change that introduced this problem?

Please take a look: https://bugzilla.mozilla.org/attachment.cgi?id=9141193

Comment 18

[:Gijs (out for now; he/him)]

The autoland pushlog implied by attachment 9141193 [details] is https://hg.mozilla.org/integration/autoland/pushloghtml?fromchange=0be81664&tochange=fe3c7983 .

bug 1347710 looks most likely. Gio, if you set security.sandbox.gpu.level to 0 in about:config (and restart), does that fix the issue?

Comment 19

[Giovanni [:Gioxx] (Mozilla Italia)]

(In reply to :Gijs (he/him) from comment #18)

bug 1347710 looks most likely. Gio, if you set security.sandbox.gpu.level to 0 in about:config (and restart), does that fix the issue?

With security.sandbox.gpu.level set to 0 all works fine and now I can scroll the page using mouse wheel.

Comment 20

[:Gijs (out for now; he/him)]

Thanks for all your help, Gio!

Comment 21

[Giovanni [:Gioxx] (Mozilla Italia)]

You're welcome :-)

Comment 22

[Davyd McColl]

setting security.sandbox.gpu.level to 0 resolves for me too. Should I be resetting this at some point?

Comment 23

[:Gijs (out for now; he/him)]

(In reply to Davyd McColl from comment #22)

setting security.sandbox.gpu.level to 0 resolves for me too. Should I be resetting this at some point?

That shouldn't be necessary. When nightlies appear for this morning, the pref's default value will go back to 0 (we're backing out the breaking change that changed it from 0 to 1) and when your prefs file is next saved, the custom/user-set value will disappear because it is identical to the default. It'd be worth double-checking that that's happened (ie that the value shows up as "default", ie not in bold type, in about:config) in a day or two, though. :-)

Comment 24

[Bob Owen (:bobowen)]

I'm guessing we're using and HWND or something that we no longer have access to or windows messages are not be delivered in the same way.

Seems like the pain is going to be reproducing on one of your machines.

Comment 25

[Firefox Bug Husbandry Bot]

Resetting severity to default of --.

Comment 26

[Chris Martin [:cmartin]]

I've tried reproducing the issue on 4 machines now, and 2 of them were even laptops with similar hardware based on the log that :Gioxx shared (thanks!). Unfortunately, I've had no luck reproducing it :(.

I wonder if it could be a software issue? I notice that Symantec Endpoint Protection is on :Gioxx 's laptop. Virus scanners do sometimes interfere with window messages and do strange things with window hooking.

Could I please ask :Gioxx to try disabling Symantec Endpoint Protection (if possible), set the "security.sandbox.gpu.level" pref to "1", restart Firefox, and see if the scrolling issue still exists?

Comment 27

[Gian-Carlo Pascutto [:gcp]]

Francesco, can you ask your friend for his about:support? We can't reproduce this and the way forward isn't very clear. We'd like to find a commonality between the people who ran into this.

Comment 28

[Francesco Lodolo [:flod]]

(In reply to Gian-Carlo Pascutto [:gcp] from comment #27)

Francesco, can you ask your friend for his about:support? We can't reproduce this and the way forward isn't very clear. We'd like to find a commonality between the people who ran into this.

That's the person already NI (we overlapped in the first comment).

Comment 29

[Augusto Pace]

Hi,

Unfortunately I'm not able to reproduce this issue. I'm setting component to Firefox - General. Adding myself as cc.

Comment 30

[Giovanni [:Gioxx] (Mozilla Italia)]

Sorry for the delay.

(In reply to Chris Martin [:cmartin] from comment #26)

Could I please ask :Gioxx to try disabling Symantec Endpoint Protection (if possible), set the "security.sandbox.gpu.level" pref to "1", restart Firefox, and see if the scrolling issue still exists?

SEP disabled, setting modified and Firefox restarted. Same problem, I can't scroll webpage.

Comment 31

[Chris Martin [:cmartin]]

(In reply to Giovanni [:Gioxx] (Mozilla Italia) from comment #30)

Sorry for the delay.

(In reply to Chris Martin [:cmartin] from comment #26)

Could I please ask :Gioxx to try disabling Symantec Endpoint Protection (if possible), set the "security.sandbox.gpu.level" pref to "1", restart Firefox, and see if the scrolling issue still exists?

SEP disabled, setting modified and Firefox restarted. Same problem, I can't scroll webpage.

Thanks for all the help. We've tried to crowd-source the reproduction effort and haven't had much luck so far, so you're one of our only connections to this bug.

Could I please ask you to try 2 more pref changes to see if it changes anything?

For both of these, make sure that "security.sandbox.gpu.level" is set to "1" (which should break scrolling), and then make the following changes separately:

1. Set "gfx.webrender.force-disabled" to "true", restart Firefox, and observe if scrolling is fixed. Set the pref back to "false" when done.
2. Set "layers.acceleration.disabled" to "true", restart Firefox, and observe if scrolling is fixed. Set the pref back to "false" when done.

Thanks!

Comment 32

[Giovanni [:Gioxx] (Mozilla Italia)]

I set to 1 security.sandbox.gpu.level and:

• with "gfx.webrender.force-disabled" set on true the bug still remain after Firefox reboot. I have resetted the option and restarted again Firefox.
• with "layers.acceleration.disabled" set on true I can scroll webpages (and security.sandbox.gpu.level is still set to 1) :-)

Giovanni.

Comment 33

[Chris Martin [:cmartin]]

(In reply to Giovanni [:Gioxx] (Mozilla Italia) from comment #32)

I set to 1 security.sandbox.gpu.level and:

• with "gfx.webrender.force-disabled" set on true the bug still remain after Firefox reboot. I have resetted the option and restarted again Firefox.
• with "layers.acceleration.disabled" set on true I can scroll webpages (and security.sandbox.gpu.level is still set to 1) :-)

Giovanni.

Glad that it worked with the SW path. That provides an important hint :)

Could I please get you to perform the following instructions to send some more detailed scroll-specific diagnostics. This will help figure out if, for example, touchpad drivers might be interfering with the HW accelerated input window.

IMPORTANT NOTE You must close all Firefox windows for all Nightly and Release installations and keep them closed except as detailed below. Every time Nightly or Release starts up, it clobbers whatever logs are in the log directory. I advise taking these instructions and copying them to Notepad to follow along. You can use either Nightly or the 75.0 release for this test.

1. Start with the "security.sandbox.gpu.level" pref set to default of "0" (scrolling works)
2. Open the Start Menu and type "environment" and select "Edit environment variables for your account"
3. The panel that's open will have a "User variables for <username>" section and a "System variables" section. We will deal with the first section
4. Click "New..." for the "User variables". Name the variable "MOZ_LOG" and enter the value "sync,MouseScrollHandlerWidgets:5" (without the quotes)
5. Click "New..." again. Name the variable "MOZ_LOG_FILE" and the value should point to the name of the log file that you want. I personally recommend that you create a new directory and name the file "mozlog" within that directory. Example: "C:\Users\cmartin\Desktop\logs\mozlog" implies that the "Desktop\logs" directory already exists, and that the output logs will be named "mozlog.xxx"
6. Open Firefox, scroll around and observe that it's working
7. Close Firefox, zip the contents of your log directory somewhere on your computer
8. Open Firefox, goto "about:config", set "security.sandbox.gpu.level" pref to "1" and close Firefox
9. Delete the contents of your log directory
10. Open Firefox, scroll around and observe that it's broken
11. Close Firefox, zip the contents of your log directory somewhere on your computer
12. You can now delete the "MOZ_LOG" and "MOZ_LOG_FILE" environment vars you added earlier, and set "security.sandbox.gpu.level" back to "0", and delete the log directory you created
13. Attach both zip files to this Bugzilla ticket

Thanks again for the help :)

Comment 34

[Giovanni [:Gioxx] (Mozilla Italia)]

Attachment: security.sandbox.gpu.level_0.zip

Comment 35

[Giovanni [:Gioxx] (Mozilla Italia)]

Attachment: security.sandbox.gpu.level_1.zip

Comment 36

[Giovanni [:Gioxx] (Mozilla Italia)]

Ciao Chris, you cand find all logs attached to this bug. Obviously security.sandbox.gpu.level_0.zip contains log with security.sandbox.gpu.level disabled and security.sandbox.gpu.level_1.zip with security.sandbox.gpu.level enabled.

Comment 37

[Chris Martin [:cmartin]]

(In reply to Giovanni [:Gioxx] (Mozilla Italia) from comment #36)

Ciao Chris, you cand find all logs attached to this bug. Obviously security.sandbox.gpu.level_0.zip contains log with security.sandbox.gpu.level disabled and security.sandbox.gpu.level_1.zip with security.sandbox.gpu.level enabled.

Ciao, Giovanni!

Thank you for the logs. They have helped a lot to narrow down potential issues. I think I may know what specific part of Firefox is to blame. Could I please get you to test another pref combination to confirm? This one is pretty simple:

Open "about:config" and set the following prefs:
"security.sandbox.gpu.level" = "1"
"gfx.webrender.force-disabled" = true
"gfx_direct3d11_use_double_buffering" = false

NOTE That last pref may-or-may-not already exist. If it doesn't, you will need to make sure "boolean" is selected and hit the little "+" sign to add it. Its value should already be set to "true" once it is created, and you should still change it to "false".

Restart Firefox with these prefs set, and if scrolling is fixed then I think we found the culprit.

Comment 38

[Giovanni [:Gioxx] (Mozilla Italia)]

(In reply to Chris Martin [:cmartin] from comment #37)

Open "about:config" and set the following prefs:
"security.sandbox.gpu.level" = "1"
"gfx.webrender.force-disabled" = true
"gfx_direct3d11_use_double_buffering" = false

Unfortunately with this configuration I can't scroll webpages :-(

Comment 39

[Pascal Chevrel:pascalc]

Fixed in 77 by the backout in bug 1347710. 78 is not affected yet as the patch as not been relanded yet.

Comment 40

[dobry1407]

I can confirm this in newest 78.0a1 Nightly. I'm using AMD Radeon RX 5700 XT GPU. Fixed by disabling GPU sandboxm but there's something definitely wrong with it...

Comment 41

[:Gijs (out for now; he/him)]

(In reply to dobry1407 from comment #40)

I can confirm this in newest 78.0a1 Nightly. I'm using AMD Radeon RX 5700 XT GPU. Fixed by disabling GPU sandboxm but there's something definitely wrong with it...

Can you file a separate bug if you're definitely on 78? The patch that caused this bug was backed out, so it should no longer be happening on an up-to-date nightly, so what you're seeing is likely a separate bug...

Comment 42

[:Gijs (out for now; he/him)]

(In reply to :Gijs (he/him) from comment #41)

(In reply to dobry1407 from comment #40)

I can confirm this in newest 78.0a1 Nightly. I'm using AMD Radeon RX 5700 XT GPU. Fixed by disabling GPU sandboxm but there's something definitely wrong with it...

Can you file a separate bug if you're definitely on 78? The patch that caused this bug was backed out, so it should no longer be happening on an up-to-date nightly, so what you're seeing is likely a separate bug...

Oh, so it appears the patch in bug 1347710 relanded (without closing the bug?), but only for non-intel gfx cards. It'd still be useful to understand more about why this also affects AMD cards... :cmartin, can you take a look?

Comment 43

[Chris Martin [:cmartin]]

(In reply to dobry1407 from comment #40)

I can confirm this in newest 78.0a1 Nightly. I'm using AMD Radeon RX 5700 XT GPU. Fixed by disabling GPU sandboxm but there's something definitely wrong with it...

"AMD Radeon RX 5700 XT GPU"

This seems to imply that you are running on a desktop machine? Can you please confirm that you are experiencing this issue on a desktop computer that has only a regular computer mouse and no touchpad, trackpad, etc?

Could we please ask you to post your support info so we can see if there are commonalities between machines that repro this issue?

(help > troubleshooting information, then click 'copy raw data to clipboard', then paste in the attachment box at https://bugzilla.mozilla.org/attachment.cgi?bugid=1630860&action=enter ) ?

Comment 44

[Gian-Carlo Pascutto [:gcp]]

If anyone else is experiencing this bug, this is one of the few cases where we'd be more than happy for people to pile in with me-too's and their about:support info. Thanks!

Comment 45

[Gian-Carlo Pascutto [:gcp]]

Davyd, as you experienced this bug originally (and may have run into it again now!), we'd very much value your about:support info.

Comment 46

[dobry1407]

Attachment: about:support raw data

This is indeed my desktop PC with regular mouse and keyboard. My about:support info is in the attachment.

Comment 47

[Davyd McColl]

Attachment: about:support from davydm

Comment 48

[Davyd McColl]

Gian-Carlo: sure, no problem... attached. I haven't run into this problem again since setting security.sandbox.gpu.level to 0

Comment 49

[BugBot [:suhaib / :marco/ :calixte]]

The severity field is not set for this bug.
:gcp, could you have a look please?

For more information, please visit auto_nag documentation.

Comment 50

[Chris Martin [:cmartin]]

Hi there folks,

Sorry that it's been a while since I've had a chance to follow up on this bug.

In order to move forward on fixing it, I have created a special build of Firefox with verbose mouse wheel logging in it. As always, we'd appreciate your help fixing this issue by gathering logs using the following instructions:

WARNING While this special Firefox is running, all movements of your mouse will be logged. (The keyboard will not be logged) Be sure this special Firefox is not open before you do anything where your mouse movements could compromise security. Specifically, DO NOT enter a password through an on-screen keyboard using either a mouse or a touchscreen while this build is running.

1. Download the "target.zip" file from the Firefox CI build server
2. It contains a single "firefox" directory. Extract it to somewhere convenient (such as your desktop or a temporary folder)
3. Open the "firefox" folder and run the "firefox.exe" file within
4. Navigate to your favourite scrollable page and scroll around using the mouse wheel
5. Expected result above: The mouse wheel should be working properly, and there will now be a file on your desktop named "firefox_scroll_trace_sandbox_disabled.txt"
6. Navigate to "about:config" and set the "security.sandbox.gpu.level" pref to '1'.
7. Close and re-open the browser
8. Again navigate to your favourite scrollable page and scroll around using the mouse wheel
9. Expected result: The mouse wheel is now broken, and there will be a file on your desktop named "firefox_scroll_trace_sandbox_enabled.txt"
10. Close the Firefox window
11. Attach the 2 files on your desktop named "firefox_scroll_trace_sandbox_disabled.txt" and "firefox_scroll_trace_sandbox_enabled.txt" to the Bugzilla ticket

Thanks!

Comment 51

[Davyd McColl]

@cmartin no worries, things have been a little shaken up of late, even more-so in mozilla-land ):

I haven't experienced this issue on Firefox Nightly in quite some time; I'm happy to mark as (perhaps accidentally) resolved.

Comment 52

[Giovanni [:Gioxx] (Mozilla Italia)]

Ciao Chris,
this is so strange. If I use my Nightly - 82.0a1 (2020-08-31) (64 bit) - with a clean profile I can reproduce the mouse wheel problem.
If I use your Firefox and I change the security.sandbox.gpu.level option in about:config the scroll works correctly after Firefox restart.

I attach the two log files produced.
Bye!
Giovanni.

Comment 53

[Giovanni [:Gioxx] (Mozilla Italia)]

Attachment: firefox_scroll_trace_sandbox_disabled.txt

Comment 54

[Giovanni [:Gioxx] (Mozilla Italia)]

Attachment: firefox_scroll_trace_sandbox_enabled.txt

Comment 55

[Davyd McColl]

@cmartin I'd forgotten about the workaround, so I tried a clean profile, and I didn't have the problem, but I also see that security.sandbox.gpu.level was set to zero -- is that default now? Or should I re-test with another value?

Comment 56

[Gian-Carlo Pascutto [:gcp]]

Davyd, see the instructions here: https://bugzilla.mozilla.org/show_bug.cgi?id=1630860#c50

We indeed set the feature to zero in normal Firefox builds in order for the scroll wheel to keep working. The instructions tell you when and were it should be re-enabled for the test build with logging.

Comment 57

[Chris Martin [:cmartin]]

(In reply to Davyd McColl from comment #51)

@cmartin no worries, things have been a little shaken up of late, even more-so in mozilla-land ):

I haven't experienced this issue on Firefox Nightly in quite some time; I'm happy to mark as (perhaps accidentally) resolved.

(In reply to Davyd McColl from comment #55)

@cmartin I'd forgotten about the workaround, so I tried a clean profile, and I didn't have the problem, but I also see that security.sandbox.gpu.level was set to zero -- is that default now? Or should I re-test with another value?

Hi Davyd,

Thanks for being so responsive :)

We changed the default for security.sandbox.gpu.level back to '0' because of the issue you, Giovanni, and dobry1407 were having above when we changed the default to '1'. But our long-term goal is to make the default '1', as it will improve Firefox security. We can't do that until we can guarantee that '1' will work properly on everyone's computer.

The instructions I wrote above in Comment 50 will collect logs with both '0' and '1' so we can compare the two and see why mouse scrolling isn't being detected with '1'. The more logs, the better -- So we'd appreciate the help if you would run through those directions :)

Cheers!

Comment 58

[Chris Martin [:cmartin]]

(In reply to Giovanni [:Gioxx] (Mozilla Italia) from comment #52)

Ciao Chris,
this is so strange. If I use my Nightly - 82.0a1 (2020-08-31) (64 bit) - with a clean profile I can reproduce the mouse wheel problem.
If I use your Firefox and I change the security.sandbox.gpu.level option in about:config the scroll works correctly after Firefox restart.

I attach the two log files produced.
Bye!
Giovanni.

Ciao Giovanni,

As always, thank you for the thorough testing and detailed info. It was smart that you did the comparison with Nightly :)

I must say this is a very interesting result! The build I linked should basically be "32-bit Nightly + Extra scroll wheel logging".

So I see 3 possibilities:

1. From your previous logs, I see you were using the 64-bit Nightly. The link above was for a 32-bit custom build. Perhaps the 32-bit and 64-bit builds behave differently?
2. Perhaps the extra scroll wheel telemetry that I added somehow "fixed" the scroll wheel?
3. Perhaps the custom builds that I'm linking are actually different from Nightly in a way I didn't realize

To help narrow these possibilities down, could I get you to try the 2 custom builds below and see if the scroll wheel breaks with them? (There should be no need to edit security.sandbox.gpu.level in either of these builds because I set the default to '1', although you can verify this in about:config if you're feeling cautious)

Build 1: This is the same build I linked above with the scroll wheel telemetry, but the 64-bit version of it (the one I sent up above was 32-bit - Perhaps it makes a difference?)

Build 2: This is a 64-bit build without any of my changes at all. This should be almost identical to Nightly, so I expect that scrolling will be broken. If it is not, it might mean that Mozilla's custom builds are not as similar to Nightly as I thought.

Thanks, and looking forward to seeing what you find!

Comment 59

[Giovanni [:Gioxx] (Mozilla Italia)]

Ok Chris, this are the results:

• Firefox 82.0a1 (2020-09-01) (64 bit), NEW PROFILE, security.sandbox.gpu.level manually set to 1: mouse wheel is broken.
• Firefox Build 1, NEW PROFILE, security.sandbox.gpu.level already set to 1: mouse wheel works correctly.
• Firefox Build 2, NEW PROFILE, security.sandbox.gpu.level already set to 1: mouse wheel works correctly.

Ok, my head is going to explode! :-D

In attachment you can find firefox_scroll_trace_sandbox_enabled.txt.

Comment 60

[Giovanni [:Gioxx] (Mozilla Italia)]

Attachment: firefox_scroll_trace_sandbox_enabled.txt

Comment 61

[Gian-Carlo Pascutto [:gcp]]

David, can you think of any difference between the two builds (Nightly vs try build from m-c) that could explain a behavior difference like this?

Comment 62

[(Away)]

The try builds from comment 58 are only opt so they lack the PGO and LTO from shippable builds.

I've triggered a shippable build on https://treeherder.mozilla.org/#/jobs?repo=try&group_state=expanded&revision=825501363431588a635f28826425c5e8e4918c9b so that we can compare apples to apples.

Comment 63

[Chris Martin [:cmartin]]

Thanks, dmajor!

(In reply to Giovanni [:Gioxx] (Mozilla Italia) from comment #59)

Ok, my head is going to explode! :-D

That makes two of us! 🤣 We've got ourselves a really tricky bug here, eh?

Could I ask you to test using the build that :dmajor just made above? Here is the link to the target.zip artifact from it. It already has the security.sandbox.gpu.level defaulted to '1'.

This will show us if there is some difference between the Nightly and "Opt" build that I sent you before.

Cheers, and as always thanks for your patience with this whole thing!
Chris

Comment 64

[Giovanni [:Gioxx] (Mozilla Italia)]

• Firefox Build 3, NEW PROFILE, security.sandbox.gpu.level verified (already set to 1): mouse wheel works correctly.

I'm going to attach the usual log file.

(In reply to Chris Martin [:cmartin] from comment #63)

Cheers, and as always thanks for your patience with this whole thing!

No problem, I'm happy to cooperate ;-)

Comment 65

[Giovanni [:Gioxx] (Mozilla Italia)]

Attachment: firefox_scroll_trace_sandbox_enabled.txt

Comment 66

[Chris Martin [:cmartin]]

Ciao Giovanni,

Sorry about the delay - I was off work Thursday+Friday, and Monday was a public holiday in Canada.

Firefox Build 3, NEW PROFILE, security.sandbox.gpu.level verified (already set to 1): mouse wheel works correctly.

This is interesting, as it should be almost identical to a Nightly build -- But this works and Nightly is broken.

I wonder if the difference is perhaps caused by "installed" versus "extracted from a .zip file and run on your desktop"?

I have an idea of how to test this. This link is a .zip file containing the latest Firefox 80.0.1 (64-bit) Release.

Could I please request that you extract the "firefox" directory to your desktop and run the "firefox.exe" in there, set security.sandbox.gpu.level to '1', restart the browser, and check if your mouse scrolling works?

For comparison, could I also ask you to test the regular Firefox release that you would normally download and install from the Mozilla website to make sure it's still broken when you set security.sandbox.gpu.level to '1'?

These 2 things should be identical except for the installation, so if your scroll wheel works then that narrows down the problem.

Thanks!
Chris

Comment 67

[Giovanni [:Gioxx] (Mozilla Italia)]

Ciao Chris,

(In reply to Chris Martin [:cmartin] from comment #66)

Could I please request that you extract the "firefox" directory to your desktop and run the "firefox.exe" in there, set security.sandbox.gpu.level to '1', restart the browser, and check if your mouse scrolling works?

Yes, the mouse scrolling works with this build and security.sandbox.gpu.level set to 1 (and - obviously- a new profile created from scratch). Firefox is executed from my Downloads folder.

For comparison, could I also ask you to test the regular Firefox release that you would normally download and install from the Mozilla website to make sure it's still broken when you set security.sandbox.gpu.level to '1'?

Bingo. Firefox Stable updated to 80.0.1 (64 bit) ("C:\Program Files\Mozilla Firefox\firefox.exe"), security.sandbox.gpu.level set to 1, restarted, mouse scrolling is now broken.

Comment 68

[Gian-Carlo Pascutto [:gcp]]

Toshihito, it looks like this might be caused by third party software hooking only files in specific dirs. What's the easiest way to find the offending module? Have the user intentionally crash Firefox and paste the about:crashes link here?

Comment 69

[Toshihito Kikuchi [:toshi]]

(In reply to Gian-Carlo Pascutto [:gcp] from comment #68)

Toshihito, it looks like this might be caused by third party software hooking only files in specific dirs. What's the easiest way to find the offending module? Have the user intentionally crash Firefox and paste the about:crashes link here?

Having a crash report is powerful, but it may be overkill for now.

To dump all loaded modules in a process, I think Get-Process Cmdlet of PowerShell is a good option. For example, running the following command in a privileged PowerShell saves all loaded modules in firefox.exe as a JSON. This combines results from all processes into one output, but I think there's no problem here. We can compare two JSONs: one from an installed firefox, the other in a copied folder.

PS> Get-Process -Name firefox -Module | ConvertTo-Json | Out-File D:\MSWORK\ff-modules.json -Encoding utf8

Another option is to use Process Hacker. It is similar to Task Manager or Process Explorer, but it has the Modules tab in the process detail dialog, meaning we can see loaded modules graphically, but I cannot find a way to export them.

Comment 70

[Giovanni [:Gioxx] (Mozilla Italia)]

Ciao Gian-Carlo, ciao Toshihito.
In attachment you can find the export from PowerShell with Firefox installed and Firefox "portable" (the zipped version I have in the Downloads folder), both with security.sandbox.gpu.level set to 1.
Have a nice evening.

Comment 71

[Giovanni [:Gioxx] (Mozilla Italia)]

Attachment: ff-installed_security.sandbox.gpu.level.1.json

Comment 72

[Giovanni [:Gioxx] (Mozilla Italia)]

Attachment: ff-portable_security.sandbox.gpu.level.1.json

Comment 73

[Chris Martin [:cmartin]]

(Nevermind - I had written instructions here, but I see that Giovanni already replied with the needed data)

Comment 74

[Chris Martin [:cmartin]]

Ciao Giovanni,

Awesome that you were able to get us those module dumps.

A few of us compared them, and the only really obvious difference is that Symantec Endpoint Protection is injecting its hooking .dll file, IPSEng64.dll, into the Firefox running out of your Program Files directory, but not the one from the .zip file.

I know we already went through this above when we asked you to disable Symantec Endpoint Protection, but perhaps disabling it isn't enough to stop it from injecting that .dll into Firefox?

So, we had three ideas:

1. Could we ask you to disable Symantec Endpoint Protection (SEP) again, and then run that same PowerShell command above and send us the .json to see if the IPSEng64.dll is still injected even with SEP disabled?
2. If you are willing, could you try completely uninstalling SEP (temporarily) and seeing if Firefox works with security.sandbox.gpu.level set to '1'? (It's totally okay if you are not comfortable with this -- No pressure, we can try other things but this is obviously a very direct route)
3. I have created a build with installer that tries to block the DLL injection even with SEP enabled. It should already have security.sandbox.gpu.level set to '1'. If scrolling works in this build, this might be a solution to this problem altogether!
   WARNING The default install directory for this is "<Program Files>\Firefox Nightly", but I'd suggest you name it something else like "<Program Files>\Firefox Custom" so you don't overwrite your Nightly build. It must be installed underneath "Program Files" though.

Thanks, as always. Hopefully we are close to the finish line here!
Chris

Comment 75

[Giovanni [:Gioxx] (Mozilla Italia)]

Ciao Chris :-)

(In reply to Chris Martin [:cmartin] from comment #74)

1. Could we ask you to disable Symantec Endpoint Protection (SEP) again, and then run that same PowerShell command above and send us the .json to see if the IPSEng64.dll is still injected even with SEP disabled?

I've tried to disable SEP but I can't, and I don't know why. We have updated to a new version of SEP in my company and I think there is a problem with the "smc -stop" command, but this is another argument and I will analyze with my colleagues in IT dept.

3. I have created a build with installer that tries to block the DLL injection even with SEP enabled. It should already have security.sandbox.gpu.level set to '1'. If scrolling works in this build, this might be a solution to this problem altogether!
   WARNING The default install directory for this is "<Program Files>\Firefox Nightly", but I'd suggest you name it something else like "<Program Files>\Firefox Custom" so you don't overwrite your Nightly build. It must be installed underneath "Program Files" though.

Unfortunately no, this release doesn't solve the problem. I have installed in %ProgramFiles%\FxTest with a new profile.
You can find the json attached to the bug, IPSEng64.dll is not in the list but mouse scroll is broken.

Comment 76

[Giovanni [:Gioxx] (Mozilla Italia)]

Attachment: NightlyTest.json

Comment 77

[Gian-Carlo Pascutto [:gcp]]

Now that we know the install location is what determines what makes things work or not, does it make sense to go back and run the logging builds (presumably throwing the logging installation in Program Files would cause it to have broken scrolling)?

If we know where the messages are getting lost, could that lead us back to the culprit?

FWIW the lastest log has scksp.dll and WinSCard.dll loaded, but those are Microsoft DLL's for smartcards so I don't think they're related to the problem.

Comment 78

[Gian-Carlo Pascutto [:gcp]]

Could it be that something expects to be able to load a DLL or file in the GPU process, but due to the changed permissions is suddenly no longer able to do that from Program Files, and that breaks scrolling? In this case the list of modules might have the "bad" one the other way around from what we expect. I didn't immediately see anything like it in the diff yesterday, but I was looking more in the other direction...

Comment 79

[Chris Martin [:cmartin]]

Ciao Giovanni,

I suspect Gian-Carlo is right, and it's something about the security permissions of Program Files causing the issue.

A few of us also noted that there is a difference between ff-installed and ff-portable regarding what DirectX .dlls are loaded. Perhaps something about the permissions is causing the loading to fail?

Here are a few experiments that might help to determine the root cause:

1. If you just download one of the .zip files, extract it, and then cut-paste the folder into "Program Files", does it break scrolling?
2. Using the official Mozilla Firefox release, could you run the above PowerShell command with security.sandbox.gpu.level set to '0' and '1' and attach both .json files?
3. Here is a link to an installer for the same custom build with the mouse logging from Comment 50. Could we ask that you install it, run it with security.sandbox.gpu.level set to '0' and '1', and send the "firefox_scroll_trace_sandbox_enabled.txt" and "firefox_scroll_trace_sandbox_disabled.txt" files that will appear on your desktop?

Sorry that this is getting more involved, but thanks for sticking with us so far!

Cheers!
Chris

Comment 80

[Giovanni [:Gioxx] (Mozilla Italia)]

Ciao Chris,

1. If you just download one of the .zip files, extract it, and then cut-paste the folder into "Program Files", does it break scrolling?
   You're right. If I download the zip file from https://bugzilla.mozilla.org/show_bug.cgi?id=1630860#c66 and I extract the "firefox" folder in %ProgramFiles% I can reproduce the bug using security.sandbox.gpu.level = 1 on a new profile.

2. Using the official Mozilla Firefox release, could you run the above PowerShell command with security.sandbox.gpu.level set to '0' and '1' and attach both .json files?
   Attached.

3. Here is a link to an installer for the same custom build with the mouse logging from Comment 50. Could we ask that you install it, run it with security.sandbox.gpu.level set to '0' and '1', and send the "firefox_scroll_trace_sandbox_enabled.txt" and "firefox_scroll_trace_sandbox_disabled.txt" files that will appear on your desktop?
   Attached.

Comment 81

[Giovanni [:Gioxx] (Mozilla Italia)]

(Sorry for the wrong formatting)

Comment 82

[Giovanni [:Gioxx] (Mozilla Italia)]

Attachment: firefox_scroll_trace_sandbox_disabled.txt

Comment 83

[Giovanni [:Gioxx] (Mozilla Italia)]

Attachment: firefox_scroll_trace_sandbox_enabled.txt

Comment 84

[Giovanni [:Gioxx] (Mozilla Italia)]

Attachment: security.sandbox.gpu.level.0.json

Comment 85

[Giovanni [:Gioxx] (Mozilla Italia)]

Attachment: security.sandbox.gpu.level.1.json

Comment 86

[Chris Martin [:cmartin]]

Ciao Giovanni,

It's very interesting that just copying the folder into Program Files is enough to break the scrolling. The plot thickens! 🤣

It seems like there are still 2 possibilities:

1. It has to do with the permissions on Program Files that are inherited by the firefox directory
2. Something in the system is treating the "Program Files" directory special, likely based on its name

I have an idea of how to narrow this down -- We can create a directory with the same permissions as Program Files, but a different name.

Here are the steps:

1. Create a directory in your "C:". I'll call it "GioxxTest"
2. Extract the .zip file for the official Mozilla Firefox into "C:\GioxxTest" (Now FF will be in "C:\GioxxTest\firefox" directory)
3. Now we need to change "C:\GioxxTest" so it has the exact same permissions as "C:\Program Files"
4. Open an administrator command prompt
5. Run icacls "c:\Program Files" /save %USERPROFILE%\Desktop\perms.txt
6. There is now a "perms.txt" on your desktop
7. Open the file in a text editor
8. Change the first line from "Program Files" to "GioxxTest"
9. Run icacls c:\ /restore %USERPROFILE%\Desktop\perms.txt
10. Now run firefox out of the GioxxTest directory
11. Make sure to set security.sandbox.gpu.level to '1'

If the scrolling is broken, that likely means it's a permissions issue. If it works, that probably means it's some kind of special treatment of Program Files.

This should help me know whether I should be looking for nefarious other processes interfering vs Windows itself just not liking the things that we're doing.

Thanks, and have a great weekend! :)
Chris

Comment 87

[Giovanni [:Gioxx] (Mozilla Italia)]

And here we are Chris :-)
Downloaded Firefox stable from Mozilla, opened with 7-Zip and extracted the "core" folder on C:\FxTest. Icacls executed to copy / restore permission, Firefox started with new profile and I have modified security.sandbox.gpu.level from 0 to 1.
Restarted Firefox and the mouse wheel is broken :-)

Comment 88

[Chris Martin [:cmartin]]

Ciao Giovanni,

It took me a while to get this working, but I have created a tool that will hopefully give enough information to answer the question, "If Firefox is not receiving the scroll wheel messages from Windows, then where are they going?"

I will attach a .zip file with a special tool I wrote to follow mouse messages around the system to see where they are dispatched.

Please unzip the contents and run the "mouselogger.exe" file within (You will have to accept the UAC privilege warning). Leave the console window open the entire time, and run the standard Mozilla Firefox from mozilla.org

Please do a bunch of scrolling in the Firefox window with security.sandbox.gpu.level = 0 (the default) and security.sandbox.gpu.level = 1.

When you are done, press "enter" button in the "mouselogger.exe" console window to close the utility.

Finally, attach the "mouselog.txt" from your desktop to this Bugzilla ticket.

Thanks, and hopefully this can help a lot to figure out where the Windows messages are going.

Comment 89

[Chris Martin [:cmartin]]

Attachment: mouselogger.zip

This is the attached .zip file.

Please note that I have already built mouselogger.exe for you. If the idea of running a privileged executable compiled on someone else's computer makes you nervous, I have included the source code and a build script so you can build it yourself.

If you want to do that, you need to have Visual Studio 2017 or higher installed with the "Desktop C++ Development" workload. You can get the free community edition from this website

Once Visual Studio is installed, simply run the "build.cmd" file included in the .zip to re-compile all binaries.

If you are not comfortable with either of these 2 options (Using the one I compiled for you or compiling it yourself), I can arrange for a Mozilla CI build machine to build it for you. :)

As always, thanks for the help!

Comment 90

[Giovanni [:Gioxx] (Mozilla Italia)]

Ciao Chris and sorry for the delay. I've attached the log.
Hace a nice weekend.
Giovanni.

Comment 91

[Giovanni [:Gioxx] (Mozilla Italia)]

Attachment: mouselog.txt

Comment 92

[Chris Martin [:cmartin]]

Ciao Giovanni,

The log that is attached strangely has zero messages about the mouse wheel going to the correct Firefox window, even with security.sandbox.gpu.level = 0.

That is very unexpected. Could I please ask you to double-check a couple of things:

1. When you open the mouselogger.exe file, it should pop up a "User Account Control" dialog box. If that doesn't happen, the file may have been built wrong.
2. Please ensure that you closed all Firefox windows in between testing the scrolling with security.sandbox.gpu.level = 0 and security.sandbox.gpu.level = 1. If you leave even 1 Firefox window open (even in Private Mode) it will not reload that value.

If we still end up with nothing but A low level mouse wheel occurred messages in the log, I may have to rethink my strategy.

Thanks as always! 😅

Comment 93

[Giovanni [:Gioxx] (Mozilla Italia)]

Ciao Chris,
I had noticed the log and honestly thought it was useless!

(In reply to Chris Martin [:cmartin] from comment #92)

1. When you open the mouselogger.exe file, it should pop up a "User Account Control" dialog box. If that doesn't happen, the file may have been built wrong.
   Yes, this thing happened. I have confirmed the UAC prompt to proceed.

2. Please ensure that you closed all Firefox windows in between testing the scrolling with security.sandbox.gpu.level = 0 and security.sandbox.gpu.level = 1. If you leave even 1 Firefox window open (even in Private Mode) it will not reload that value.
   Of course, already done that.

Anyway, I have tried once again and unfortunately the result is the same :-(
I have used Firefox stable (extracted from the installer and used as portable) and also the new Firefox Nighty (84, installed on my machine as default browser).

Have a nice day!
Giovanni.

Comment 94

[Chris Martin [:cmartin]]

Since the future of GPU process sandboxing is uncertain, and this bug may no longer be relevant to whatever solution we do end up with, I'm going to close this bug. If we attempt this again in the future, we will deal with it through a different bug at that point.

Comment 95

[Chris Martin [:cmartin]]

There has been a bit of interest in trying to enable the GPU sandbox, and we are just trying to get a feel for if this issue still exists or if it was indirectly fixed by some of the other work that's been done in the last couple of years (yikes I can't believe it's been that long!)

Ciao Giovanni -- This may be a long shot, but is there any chance that you still have the piece of hardware that you last were able to reproduce this issue on?

If so, could please we ask you to try the same thing that you did a couple years ago? Open about:config, set security.sandbox.gpu.level = 1, close all Firefox windows, re-open Firefox, and check to see if mouse scrolling is still broken.

Thanks!

Comment 96

[Giovanni [:Gioxx] (Mozilla Italia)]

(In reply to Chris Martin [:cmartin] from comment #95)

There has been a bit of interest in trying to enable the GPU sandbox, and we are just trying to get a feel for if this issue still exists or if it was indirectly fixed by some of the other work that's been done in the last couple of years (yikes I can't believe it's been that long!)

Ciao Giovanni -- This may be a long shot, but is there any chance that you still have the piece of hardware that you last were able to reproduce this issue on?

If so, could please we ask you to try the same thing that you did a couple years ago? Open about:config, set security.sandbox.gpu.level = 1, close all Firefox windows, re-open Firefox, and check to see if mouse scrolling is still broken.

Thanks!

Ciao Chris, "welcome back!" :)
I have for a limited time the same hardware and software scenario of the "first time" (because within a few days I will be changing the Windows PC at work). I have tried to change the setting in my Nightly profile and all works well with security.sandbox.gpu.level = 1.
It's likely that some other update that has occurred over time has allowed the setting to work properly today.

I'm here if you need anything else, drop a line in case.

Comment 97

[Chris Martin [:cmartin]]

Thank you, Giovanni! That is great news. We will attempt to re-enable this feature shortly. Hopefully everything will Just Work™!

Comment 98

[Chris Martin [:cmartin]]

I'm going to close this bug for now, as we have nobody who reports still being able to reproduce it. Thanks to everyone who reported it, and especially to Giovanni, who really put in a lot of effort to help us figure out this issue 😄