1558167 - [10.15] WebRender broken on macOS 10.15 Catalina

Status: RESOLVED FIXED

(Core :: Graphics: WebRender, defect, P3)

Description

[Goffredo Marocchi]

Attachment: Screenshot 2019-06-10 at 13.17.54.png

User Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:69.0) Gecko/20100101 Firefox/69.0
Firefox for Android

Steps to reproduce:

Install macOS 10.15 Catalina on MacBook Pro mid 2018 (which was running Firefox Nightly with WebRender enabled through about:config)

Install Firefox Nightly and enabled Web Render

Browse the web

Actual results:

Text corruption, images rendering corruption

Expected results:

I expected the same HW to be able to run the same version of Firefox Nightly with the same WebRender renderer active and drawing to screen correctly

Comment 1

[:Gijs (he/him)]

Doesn't look like this needs to be sec-sensitive.

Comment 2

[Marcia Knous [:marcia]]

Hello Goffredo - I have WR enabled on my MacBook Pro running 10.15 and I don't see any corruption. We probably need more information about your hardware to try to debug this issue. Also can you confirm which about:config pref you toggled? Thanks!

Comment 3

[Darkspirit]

When everything is corrupted, please open about:support, click on "Copy text to clipboard" (second button top left), paste it into a text file and upload it here (Attach file). Thanks!

Comment 4

[Goffredo Marocchi]

(In reply to Marcia Knous [:marcia - needinfo? me] from comment #2)

Hello Goffredo - I have WR enabled on my MacBook Pro running 10.15 and I don't see any corruption. We probably need more information about your hardware to try to debug this issue. Also can you confirm which about:config pref you toggled? Thanks!

Hello, the preference is: gfx.webrender.all

HW is: attached screenshot for HW

Comment 5

[Goffredo Marocchi]

Attachment: HW info

Comment 6

[Goffredo Marocchi]

Attachment: AboutSupport for Firefox Nightly in text format

Comment 7

[Goffredo Marocchi]

Attachment: CatalinaFirefoxCorruptedRenderingWebRender_20190618.txt

Got an entire tab, page rendering included not just the browser UI, corrupted

Comment 8

[Graham Campbell]

I can confirm the same issue. Firefox 68.0b13 (64-bit). MacOS 10.15 Beta (19A487m).

Comment 9

[Kornel]

I'm seeing the same — glyphs rendered as red/green/cyan blobs, and it gets progressively worse (seconds after restart some text is ok, by the time I got to about:config it was all unreadable).

Setting gfx.webrender.force-disabled=true makes everything look normal again.

Comment 10

[Kornel]

^ Firefox nightly 69.0a1 (2019-07-08) (64-bit) on Catalina 10.15 Beta (19A501i) on MBP with Radeon Pro 560X 4 GB + Intel UHD Graphics 630 1536 MB

Comment 11

[cumanzor]

Attachment: ff-70b4-webrender-all-enabled

Additional data point:

MBP 2016 (Radeon Pro 450 2 GB), connected to an external 4K monitor. This is with the settings gfx.webrender.all enabled. All my add-ons were disabled for this test. Next attachment shows the webrender setting disabled.

Comment 12

[cumanzor]

Attachment: Screen Shot 2019-09-10 at 9.26.44 AM.png

From comment above. This is when restarting the browser in safe mode, which I assume ignores the webrender config and defaults to false (not sure about that though).

Comment 13

[Sören Hentzschel]

Attachment: screenshot

This is how it looks for me with WebRender enabled on macOS Catalina Beta on a MacBook Pro from late 2018 with Radeon Pro 560X.

Comment 14

[jsanfordgit]

Attachment: js-20190927-about-support.txt

Comment 15

[jsanfordgit]

Attachment: js-20190927-about-support.png

Comment 16

[jsanfordgit]

Attachment: js-20190927-settings.png

Comment 17

[jsanfordgit]

I also have this problem on macOS 10.15 Public Beta 9.

Resetting these two settings to their default of "false" fixes it:

gfx.webrender.all = false
gfx.webrender.enabled = false

This was really concerning until I remembered I had set the above to true

The problem showed up on the second or third restart after upgrading the OS -- the initial launch worked okay.

Attached about:support and two screenshot examples.

attachment 9097082 [details]
attachment 9097083 [details]
attachment 9097084 [details]

Comment 18

[Jeff Muizelaar [:jrmuizel]]

Dzmitry, can you file a bug with Apple about this issue?

Comment 19

[Dzmitry Malyshau [:kvark]]

Could be AMD-specific? I'm not able to reproduce this on macOS 10.15 (19A582a) with MBP 2016, Intel Iris 550 graphics.

Comment 20

[Dzmitry Malyshau [:kvark]]

Waiting for Markus to update the AMD-based MBP and reproduce locally...

Comment 22

[Liz Henry (:lizzard) (relman/hg->git project)]

Is there something we need to do here for 70 before release? Tracking for now, in case.

Comment 23

[Markus Stange [:mstange]]

(In reply to Dzmitry Malyshau [:kvark] from comment #20)

Waiting for Markus to update the AMD-based MBP and reproduce locally...

It turns out that machine does not have an AMD GPU. We still haven't identified a machine on which to try reproducing this in the Toronto office.

Comment 24

[Marcia Knous [:marcia]]

Adding Camelia to see if they can reproduce on the QA side.

Comment 25

[BugBot [:suhaib / :marco/ :calixte]]

Changing the priority to p1 as the bug is tracked by a release manager for the current beta.
See What Do You Triage for more information

Comment 26

[Jeff Muizelaar [:jrmuizel]]

(In reply to Liz Henry (:lizzard) from comment #22)

Is there something we need to do here for 70 before release? Tracking for now, in case.

No. WebRender is not enabled on Mac in 70. To run into this a user would need to gfx.webrender.all

Comment 27

[Markus Stange [:mstange]]

We have a machine now.

Comment 28

[Dzmitry Malyshau [:kvark]]

Attachment: Disable swizzling on macOS 10.15 AMD devices

Comment 29

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/2e23a503638e
Disable swizzling on macOS 10.15 AMD devices r=jrmuizel

Comment 30

[Arthur Iakab [arthur_iakab]]

https://hg.mozilla.org/mozilla-central/rev/2e23a503638e

Comment 31

[Dirkjan Ochtman (:djc)]

Assuming that the 20191009095806 nightly includes 2e23a503638e, this doesn't fix the issues for me.

Comment 32

[Dzmitry Malyshau [:kvark]]

Dirkjan, could you provide your about:support?

Comment 33

[Dirkjan Ochtman (:djc)]

Attachment: about:support for :djc

Comment 34

[cumanzor]

Attachment: about:support

@Dzmitry: Under 20191009103354, it seems to be affecting the browser chrome only, web pages are working fine with webrender.all set to true.

Comment 35

[Dzmitry Malyshau [:kvark]]

Wow, that's weird! Could you try running the artifact from a try push that completely disables the swizzling? Don't forget to ensure WR is ON, please.

Also, please provide a screenshot of how it works under 20191009103354 for you.

Comment 36

[cumanzor]

Attachment: Screen Shot 2019-10-09 at 9.28.36 AM.png

This on build 20191009103354. I'll test the artifact in a sec.

Comment 37

[cumanzor]

Attachment: Screen Shot 2019-10-09 at 9.33.53 AM.png

@Dzmitry: seems to be glitching out a bit under build 20191008171937 as well. The tab bar changes color from blue to white when switching tabs, and the tab title is a bit glitched out.

Comment 38

[Dzmitry Malyshau [:kvark]]

I can repro. Looking into it...
It's possible that Swizzling is not the only thing that's going wrong here.

Comment 39

[Dzmitry Malyshau [:kvark]]

Here is something interesting. Playing with the blob example I can see that:

1. it is reproducible, i.e. I see visual issues with the colors, and I can record the GL stream with apitrace and then replay it on Intel with no issues
2. the issue shows up in the texture upload stage - this is before we do anything with the actual swizzle configuration of texture units (!)
3. out of 4 uploaded chunks, only one that is not a power of two size gets buggy results

This leads me to the following hypothesis: the AMD bug is not caused by swizzling, it's in the texture uploads, related to how pixel unpack buffer is read. When swizzling is forcefully disabled, the driver has to convert the format (from BGRA to RGBA), and thus it's going a completely different code path that is not affected by the bug.

For an extra test, I disabled all the swizzle setting code (on texture units), and I still got an issue. That somewhat aligns with the observation that there are still visual issues even if swizzling is disabled in Gecko - because it's not the swizzling fault.

Comment 40

[BugBot [:suhaib / :marco/ :calixte]]

Bugbug thinks this bug is a regression, but please revert this change in case of error.

Comment 41

[Dzmitry Malyshau [:kvark]]

Attachment: Force 256 alignment for texture uploads on AMD on macOS

Comment 42

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/8d93170e3aaf
Force 256 alignment for texture uploads on AMD on macOS r=jrmuizel

Comment 43

[Dzmitry Malyshau [:kvark]]

Comment on attachment 9100208 [details]
Force 256 alignment for texture uploads on AMD on macOS

Beta/Release Uplift Approval Request

• User impact if declined: Some of the macOS users who force-enable WR will find it unusable.
• Is this code covered by automated tests?: Yes
• Has the fix been verified in Nightly?: No
• Needs manual test from QE?: No
• If yes, steps to reproduce:
• List of other uplifts needed: None
• Risk to taking this patch: Low
• Why is the change risky/not risky? (and alternatives if risky): Risk is low, because it is specifically written to only affect a small population of WR on macOS.
• String changes made/needed:

Comment 44

[Razvan Maries]

https://hg.mozilla.org/mozilla-central/rev/8d93170e3aaf

Comment 45

[cumanzor]

How can we track when this reaches the Beta channel?

Comment 46

[Dzmitry Malyshau [:kvark]]

The request for approval has been filed in Comment 43, you'll see the follow-up comments in here.

Comment 47

[Dzmitry Malyshau [:kvark]]

Added the bug to our list: https://github.com/servo/webrender/wiki/Driver-issues#bug-1558167---texture-uploads-require-the-byte-stride-of-256-on-amd-gpus-with-macos-1015

Comment 48

[Liz Henry (:lizzard) (relman/hg->git project)]

Dzmitry, is there some way we can verify this fix and can you describe it for QA?
This is very last minute for uplift to 70, since the release candidate build is tomorrow morning.
I'd prefer this get more bake time in nightly.

Comment 49

[Liz Henry (:lizzard) (relman/hg->git project)]

Comment on attachment 9100208 [details]
Force 256 alignment for texture uploads on AMD on macOS

Just a shade too last minute for my comfort levels as the RC build is tomorrow.

Comment 50

[Dzmitry Malyshau [:kvark]]

Liz, testing is very simple: run FF (edit: with WebRender enabled!) on macOS 10.15 with AMD GPU, the corruption in colors all over the screen would be obvious.

Comment 51

[0011101000101001]

I would just like to report using webrender on macos looks like this:

https://pixelfed.social/storage/m/9415380b19c20b948c98d7c5f1fb65710b040a9e/06cd91a92d0439818e7e88631d135e73fbf12efd/u27KRHPMHaRkBtjDGSuonxhHBGSt4xDKE1zcQIdN.png

since macos 10.15 (Catalina), using an egpu with rx580

This is a vanilla profile with: user_pref("gfx.webrender.all", true); user_pref("gfx.webrender.enabled", true);

Webrender was working just fine for me in 10.14.

Comment 52

[Dzmitry Malyshau [:kvark]]

Our detection is barebone: let is_amd_macos = cfg!(target_os = "macos") && renderer_name.starts_with("AMD");, perhaps the rx580 returns a different format of the renderer string? 0011101000101001, do you have a glxinfo so that you could attache the output here?

Comment 53

[0011101000101001]

If you point me to some info on how to get this on a mac, I can try to get this.

Comment 54

[Dzmitry Malyshau [:kvark]]

Perhaps you could just attach the "about:support" page output.
I can see that on Windows rx580 is identified by the GL driver as "Radeon RX 580 Series", although it tells nothing about macOS drivers.

Comment 55

[0011101000101001]

Attachment: about-support.txt

Sorry for the delay.

Comment 56

[Dzmitry Malyshau [:kvark]]

Sorry, I should have noticed that earlier (thanks :mstange for the hint!) - you are running the Release Firefox 70.0. The fix is in Nightly, but it's going to take it a few more weeks before it reaches Beta/Release channels.

Comment 57

[0011101000101001]

I thought it was a good idea to report this but it seems I skimmed too much in this thread sorry. So January 7th in stable probably. Good to know thanks.

Comment 58

[0011101000101001]

The nightly fix forks for me.