1638675 - Firefox 75 & 76 impacting RDP on Windows Server 2008 R2

Status: RESOLVED FIXED

(Core :: Graphics, defect, P1)

Description

[Austin [EarthBoundX5]]

User Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:76.0) Gecko/20100101 Firefox/76.0

Steps to reproduce:

1. RDP'd to a Windows Server 2008 R2 server (repeatable on multiple different Windows Server 2008 R2 instances)
2. Upgraded to Firefox 75 and subsequently 76 (possibly earlier, but no more than a version or two)
3. Opened Firefox

Actual results:

Experienced incredibly lag in both the Firefox window and the RDP session as a whole. Only closing Firefox resolves this. If I restart Firefox in Safe Mode, there are no issues; however this is repeatable on multiple instances, most without add-ons.

Expected results:

Firefox should have continued to function normally and not impacted my RDP session.

Additional Information:
Unsure of where to put this, so leaving it here. I already posted this on the Firefox help forums, and got nothing: https://support.mozilla.org/en-US/questions/1285436#answer-1311102

Posting details from that post here as well:
Ever since Firefox upgraded to version 75, using it has become impossible to use via RDP on Windows Server 2008 R2. The behavior is like Firefox completely hogs down the machine, but more likely seems to be impacting the refresh rate of the RDP session. With Firefox open, mouse begins to trail and button clicks and movement stack up and execute as much as a 5 minute delay at times. I saw a similar report, but not 100% the same, here: https://support.mozilla.org/en-US/questions/1278453

Here is what I know so far...

• Specific to Windows Server 2008 R2, tested on multiple VMs on multiple hosts
• Issue not present on Windows Server 2016, Windows 10, or Windows 7
• Size of Firefox window affects performance, leading to validate RDP refresh issue. If half the screen is Firefox vs the whole screen, the lag is lessened
• Safe Mode immediately resolves the issues
• Safe Mode with all Add-Ons re-enabled still resolves the issue
• Hardware acceleration on or off has no impact
• Downgrading to Firefox 74 resolves the issue (but this could also be because it made me a new profile)

Comment 1

[BugBot [:suhaib / :marco/ :calixte]]

Bugbug thinks this bug should belong to this component, but please revert this change in case of error.

Comment 2

[Dzmitry Malyshau [:kvark]]

Thank you for filing such a detailed bug report!
Marking as S4 for this being very specific to RDP Windows Server 2008 R2 and with an existing workaround (safe mode). Still must be very annoying!
Interestingly the second question you linked reports Safe mode to not help, contrary to your case.
There are a few things that would be helpful:

• "about:support" from that configuration
• Gecko profile from that run
• https://mozilla.github.io/mozregression/ used to find out the exact commit range that regressed it

Pinging Sotaro the Windows expert, just in case they have more immediate ideas.

Comment 3

[Sotaro Ikeda [:sotaro]]

(In reply to Austin from comment #0)

• Hardware acceleration on or off has no impact

Sorry, I do not have many ideas. Yea, about:support is very helpful.

It is weird that acceleration off does not have impact. I wonder if GPU process usage might affect to the bug. When hardware acceleration is off, GPU process normally exists. But with safe mode, GPU process is disabled. It might affect to the performance difference. GPU process could be disabled with pref "layers.gpu-process.enabled=false" in about:config.

Composition of no acceleration with GPU process was changed since Firefox 75 by Bug 1604412. I wonder if it might affect to this bug. The bug is a preparation work for enabling sandbox protections for GPU process(Bug 1347710). Though GPU process sandbox is not enabled yet.

Comment 4

[Austin [EarthBoundX5]]

Attachment: 123

Comment 5

[Austin [EarthBoundX5]]

Attachment: 1234

Comment 6

[Austin [EarthBoundX5]]

Posted details of about:support above.

Comment 7

[Austin [EarthBoundX5]]

I've had failures trying to upload the profiler output, here is a link to it from my dropbox. Please let me know if I need to try again and upload it here, I could do so from work instead.

https://www.dropbox.com/s/bo2e755bu2l7mla/Firefox%202020-05-22%2001.38%20profile.json.gz?dl=0

The profiler was run, selected default Web Developer option, opened a web page, moused over links while experiencing major lag, clicked on a link, moused around again with lots of lag, highlighting links, waiting for page to update or cursor to change, etc.

Comment 8

[Austin [EarthBoundX5]]

Attachment: RegressionTestingLog

Comment 9

[Austin [EarthBoundX5]]

app_name: firefox
build_date: 2019-11-21
build_file: C:\Users\Administrator.mozilla\mozregression\persist\2019-11-21--mozilla-central--firefox-72.0a1.en-US.win64.zip
build_type: nightly
build_url: https://archive.mozilla.org/pub/firefox/nightly/2019/11/2019-11-21-21-44-22-mozilla-central/firefox-72.0a1.en-US.win64.zip
changeset: 76e20a306bc262843ec06d59c54416131af030cb
pushlog_url: https://hg.mozilla.org/mozilla-central/pushloghtml?fromchange=aaae630f30291056f4f40bbd9e12a917309e401e&tochange=92c11f0bf14b71b70bec5351212ae237707f4a62
repo_name: mozilla-central
repo_url: https://hg.mozilla.org/mozilla-central

Comment 10

[Austin [EarthBoundX5]]

Attachment: 2020-05-22 02_23_43-Regression.png

Comment 11

[Austin [EarthBoundX5]]

I've pasted the log and build info sections of regression testing with Mozregression.

I've added a picture as well, as I wanted to add another comment. The first build tested, 2019-11-21 (Firefox 72) ran flawless, perfectly, nothing slow. However, every subsequent "good" rated build (Firefox 74 and partially 75) were all what I would consider mildly slow responsiveness. Nothing terrible, just noticeable when compared to the earlier build for sure. However, it was absolutely nothing compared to the lag experienced in all "bad" marked builds. Basically slow vs unusable.

If need be, I can do additional regression testing for the mildly slow stuff?

Comment 12

[Austin [EarthBoundX5]]

Adjusting "layers.gpu-process.enabled=false" in about:config in the current Firefox 76.0.1 completely resolves the slowness, comparable to the speeds of Firefox 72 (2019-11-21) mentioned above, and not in the range described between there and the unsuitably slow build.

Comment 13

[Darkspirit]

Thanks!
From comment 8:

2020-05-22T07:36:35.361000: DEBUG : Found commit message:
Bug 1604412 - Enable remote backbuffer GDI compositing r=jmathies,jld

Comment 14

[Chris Martin [:cmartin]]

I'm away until June 1st, but when I get back I'll investigate this.

Comment 15

[Chris Martin [:cmartin]]

I am currently setting up a machine with Windows 7 (the base for Windows Server 2008 R2) to see if I can repro using certain combinations of flags. Otherwise, it may be a bit harder for me to get my hands on a Windows Server 2008 R2 machine.

My intuition is that the use of RDP is disabling HW acceleration, causing the remote backbuffer code to kick in for the entire Firefox window. Since the remote backbuffer code updates the entire window if there is an update to any rectangle, it is likely spamming the RDP connection with huge update rectangles.

I need to repro to confirm, though.

Comment 16

[Chris Martin [:cmartin]]

Unfortunately I was unable to reproduce this issue on Windows 7, even using several modes that more closely emulate the behaviour of Windows Server 2008 R2.

I'm going to have to get a VMWare + Windows Server 2008 R2 license from IT, but first I'm going to try adding instrumentation to the code on my own machine to see if my hypothesis that SW mode spams the Windows GDI with rectangle updates is plausible.

Comment 17

[Chris Martin [:cmartin]]

I don't see any spamming of GDI drawing calls on my machine, which means I will likely have to go ahead with getting VMWare + Windows Server 2008 R2 to try and repro this issue. My machine is working as-expected and only performing rectangle updates when something in the FF window has actually changed, and it seems to properly max out around ~60fps or so.

Comment 18

[Gian-Carlo Pascutto [:gcp]]

Chris is still actively investigating this so P1.

Comment 19

[Gian-Carlo Pascutto [:gcp]]

We fixed bug 1647877 and that now shipped in Firefox 80.

Reporter, are you still seeing this problem with Firefox 80?

We can still improve a bit in bug 1654617 but it should already be much better if the cause is what we think it is.

Comment 20

[Gian-Carlo Pascutto [:gcp]]

Bug 1654617 now landed in Firefox 82. I am going to assume this is now fixed as there are no other reports. Please re-open if the issue persists with Firefox 82.

Comment 21

[BugBot [:suhaib / :marco/ :calixte]]

Since the status are different for nightly and release, what's the status for beta?
For more information, please visit auto_nag documentation.