1570736 - Webrender completely broken on Mac (10.14, Intel HD Graphics 4000)

Status: RESOLVED FIXED

(Core :: Graphics: WebRender, defect, P3)

Description

[Robin Whittleton]

Attachment: Nightly’s best attempt to render browser chrome and Wikipedia.

User Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.14; rv:70.0) Gecko/20100101 Firefox/70.0

Steps to reproduce:

Installed latest Nightly 2 days ago, restarted Firefox. This on a Macbook Air 11" 2012 on latest 10.14.6. So that means Intel i5 with HD Graphics 4000, with Webrender manually enabled.

Actual results:

Complete font and image breakage. Rendering boxes and general layout components seems to still work. See screenshots.

Expected results:

Normal display. If I turn webrender off (manual editing of prefs.js is easiest) then we’re back to normal; I’m filing this bug from that setup.

Comment 1

[Robin Whittleton]

Attachment: See also Omnibar rendering breakage

Comment 2

[Cristian Fogel [:cfogel]]

Attempted to reproduce, but was unable to get the issue to manifest with devices having Intel IRIS pro 1536, Intel IRIS plus 650.
Your issue might be limited to the graphics setup.

@Robin, any chance you could install mozregression and run a check for the regression range?

Comment 3

[Aria Beingessner [:Gankra]]

Oof, the corruption you're seeing is a classic example of us messing up the texture cache (basically everything that we cache a rasterization of like images/glyphs gets replaced with random textures). Sadly something that can happen in various different ways. Haven't seen this on any of our macs yet, though. Maybe the supplemental macos update that just came out is involved (https://support.apple.com/kb/DL2015?locale=en_US)? Do you have that one installed yet? (sadly I don't think it shows up as a difference in macos version number)

A regression range would definitely be helpful.

Comment 4

[Robin Whittleton]

Nope, this is post 10.14.6, pre supplemental update. I’ll run mozregression though and get back to you with a range.

Comment 5

[Robin Whittleton]

Mozregression came up with https://hg.mozilla.org/integration/autoland/pushloghtml?fromchange=ce76fa05c90f3f24f8db09950eadd4a8cdec9088&tochange=bc58681d24079ae0bdb65c9679455042dccf0e76 , which looks suitably suspicious :)

Comment 6

[Aria Beingessner [:Gankra]]

Awesome, thanks!

I also managed to reproduce this locally! Found an old 2012 macbook air in the office and updated it (was running macos 10.10 and firefox 56 🙀).

Comment 7

[Dzmitry Malyshau [:kvark]]

The texture cache is fine. Problem only manifests itself when sampling from it with swizzling states set up. It looks like setting those GL_TEXTURE_SWIZZLE states corrupts the other sampling parameters, likely a driver bug. Only seen on Intel 4000 so far - the same machine running on NVidia is fine (i.e. even replaying the same WR capture).

I've added it to https://github.com/servo/webrender/wiki/Driver-issues
Currently trying to narrow down the bug in https://github.com/kvark/gl-swizzle-test
As a short term solution we could also flip "allow_texture_swizzling = false" in RendererOptions when we detect Intel 4000 series GPU on MacOS.

Comment 8

[Dzmitry Malyshau [:kvark]]

Managed to reproduce it, at last, in the artificial test app. Basically, setting up the swizzle state on a texture unit breaks the result of textureSize instruction of GLSL, which WebRender relies on. I believe the best course of action is to blacklist swizzling for IvyBridge GPU family on Mac.

Comment 9

[Dzmitry Malyshau [:kvark]]

Attachment: Force disable WR swizzling on Intel 4000 on Mac

Investigation showed that on this platform the texture unit state becomes
corrupted whenever we set the non-identity swizzling (getting garbage from textureSize()).
Given no easy workaround, we disable swizzling for this GPU family on Mac, for now.

Comment 10

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/37cd393d162d
Force disable WR swizzling on Intel 4000 on Mac r=aosmond

Comment 11

[Natalia Csoregi [:nataliaCs]]

https://hg.mozilla.org/mozilla-central/rev/37cd393d162d

Comment 12

[Robin Whittleton]

Attachment: Demo of the broken UI

This patch fixes the original problem, but seems to have introduced interesting colour shifts. This appears to be a ROT13 style problem (can you tell I’m not a graphics person?) as the screenshots of the broken behaviour appear fine in the preview display of the bug upload field.

Comment 13

[Robin Whittleton]

Attachment: Mozilla.org with colour shift

Comment 14

[Dzmitry Malyshau [:kvark]]

Robin: yes, this is known and filed separately as Bug 1572843. The fix is going through the review now.

Comment 15

[Robin Whittleton]

OK, cool, I’ll VERIFY this one as fixed then.

Comment 16

[Cristian Comorasu, QA [:ccomorasu], Releae Desktop QA - Please contact [e-mail]]

Thank you, Robin!

Comment 17

[tupos9]

Macbook Air 13 Mid 2011, macbookair 4,2, Intel HD 3000 384 MB.
MacOs High Sierra 10.13.6 (17G8037)

The bug is still there. Just upgraded to firefox 70.

Comment 18

[Dzmitry Malyshau [:kvark]]

We blocklisted Intel 4000, while your device is HD 3000... Let's follow-up with blocklisting the swizzling code for this as well.

Comment 19

[Dzmitry Malyshau [:kvark]]

Attachment: Blocklist WR swizzling on HD3000

Comment 20

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/4f2ad8907c9e
Blocklist WR swizzling on HD3000 r=jrmuizel

Comment 21

[Cosmin Sabou [:CosminS]]

https://hg.mozilla.org/mozilla-central/rev/4f2ad8907c9e

Comment 22

[BugBot [:suhaib / :marco/ :calixte]]

Since the status are different for nightly and release, what's the status for beta?
For more information, please visit auto_nag documentation.

Comment 23

[Ryan VanderMeulen [:RyanVM]]

Please nominate this for Beta uplift when you get a chance.

Comment 24

[Dzmitry Malyshau [:kvark]]

Comment on attachment 9104646 [details]
Blocklist WR swizzling on HD3000

Beta/Release Uplift Approval Request

• User impact if declined: Intel HD3000 users on macOS would get broken WebRender (although they still have to opt into that)
• Is this code covered by automated tests?: No
• Has the fix been verified in Nightly?: Yes
• Needs manual test from QE?: No
• If yes, steps to reproduce:
• List of other uplifts needed: None
• Risk to taking this patch: Low
• Why is the change risky/not risky? (and alternatives if risky): It's just expanding the blocklist, not risky.
• String changes made/needed:

Comment 25

[tupos9]

Hello,

I verified and in nightly the bug has gone. So apparently the fix is working.

Comment 26

[Pascal Chevrel:pascalc (PTO until April 26)]

Comment on attachment 9104646 [details]
Blocklist WR swizzling on HD3000

Low risk, Webrender fix for a subset of our mac user-base by blocklisting a board model, approved for 71 beta 6, thanks.

Comment 27

[Dorel Luca [:dluca]]

https://hg.mozilla.org/releases/mozilla-beta/rev/7abf70086d87

Comment 28

[ckoff]

I'm currently seeing a similar issue on Firefox 71.0 with HD 5000. I initially believed it was due to bug 1586627, but after more thorough searching this bug and bug 1562462 point to the webrender issue as being more likely.

Firefox 71.0
Mojave/10.14.6
MacBook Pro (Retina, 13-inch, Late 2013) (MacBookPro11,1)
Intel Core i5 / 2.6 GHz
Memory: 16 GB
Intel HD Graphics 5100
VRAM (Dynamic, Max): 1536 MB

Comment 29

[Dzmitry Malyshau [:kvark]]

ckoff, could you please attach the "about:support" output?

Comment 30

[ckoff]

Attachment: about-support.json

Comment 31

[Dzmitry Malyshau [:kvark]]

ckoff, thank you! So this is Haswell iGPU, it's not in the block list for this feature.

Could you provide more information about what you are experiencing? we need to double-check if this is indeed the same issue. It would be highly unfortunate if we had to block-list Haswell swizzling. Please attach a screen shot and ideally a WR capture. You can get it by hitting "CTRL + SHIFT + 3" when you see the problem, it is saved in the "~/wr-capture" folder (which you'd need to pack into a file and maybe send my way via Firefox Send or any other service). Thank you in advance!

Comment 32

[Alan Jeffrey [:ajeffrey]]

Can confirm that enabling webrender in FF 74.0 on a 2012 mac mini with Intel HD Graphics 4000 produces this problem.

Conversation at #gfx: https://matrix.to/#/!wbEUPKmXGxodKqLlZL:mozilla.org/$HO8SQXRwbC5hTy25_FAouBtftVRjfIDWGWl1_vKFgv8?via=mozilla.org&via=matrix.org&via=amorgan.xyz

Comment 33

[Jessie [:jbonisteel] pls NI]

Starting to look at blocking Mac bugs. Dzmitry, do you have any theories as to was was happening here with swizzling?

Comment 34

[Dzmitry Malyshau [:kvark]]

It's hard to investigate without affected platform/hardware by hand... Something in the swizzling logic got broke. It could be that the issue Alan sees is different from previously reported one, since HD Graphics 4000 is blacklisted for swizzling.

Alan, it would help to get the output from WebRender logging:

cd webrender/examples
RUST_LOG=webrender=info cargo run --bin basic

Could you provide it from the affected machine?

Comment 35

[Alan Jeffrey [:ajeffrey]]

Attachment: Screenshot of administrator’s Mac mini at 2020-06-02 11:40:10.png

Here's a screenshot (running via RDP, so this is the best I could do).

Comment 36

[Alan Jeffrey [:ajeffrey]]

Also, to get this bug I had to force webrender to be enabled. (I am originally seeing this on Servo, which doesn't use the blocklisting).

Comment 37

[Dzmitry Malyshau [:kvark]]

Thank you!
That's the same as I'm getting on my Linux machine, and this path should be fairly solid. I wonder if we are seeing the same issue we blocklisted before.
Could you run https://github.com/kvark/gl-swizzle-test on that machine and report what you see, please?

Comment 38

[Dzmitry Malyshau [:kvark]]

Notably, the only thing that changed the relevant swizzle paths is https://phabricator.services.mozilla.com/D65596, so bug 1612941 is a candidate to be causing this regression. cc Lee

Comment 39

[Dzmitry Malyshau [:kvark]]

Here is what I found:

• https://phabricator.services.mozilla.com/D65596 does change the logic in a way that it requires ARB_texture_swizzle on GL. Previously, it was assumed to be available on desktop GL.
• macOS doesn't appear to report "ARB_texture_swizzle" at all, at least there are no reports. The relevant article implies that macOS with OpenGL 3.2 should report the extension though. Would be good to double check that.
• the combination of these factors mean that bug 1612941 broke swizzle detection on macOS. I'm fixing this.

Comment 40

[Dzmitry Malyshau [:kvark]]

Attachment: Fix GL capability check for swizzling

Comment 41

[Alan Jeffrey [:ajeffrey]]

Attachment: Screenshot of administrator’s Mac mini at 2020-06-02 15:11:37.png

gl-swizzle-test is green.

Comment 42

[Dzmitry Malyshau [:kvark]]

Thank you!
It seems to me that your hardware should fall under the blacklist already. Providing about:support would make it more evident.
Unless... the code path that doesn't swizzle is currently broken. I need to look into this more.
I guess I can try blocking the swizzling on my mac and checking how it works, that would be a good test.

Comment 43

[Alan Jeffrey [:ajeffrey]]

It is, like I said you have to force-enable webrender to get this behaviour, this is more of a Servo issue in that we don't have a blocklist/allowlist for individual gfx cards.

Comment 44

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/ad1dfa61b8f5
Fix GL capability check for swizzling r=lsalzman

Comment 45

[Bogdan Tara[:bogdan_tara | bogdant]]

https://hg.mozilla.org/mozilla-central/rev/ad1dfa61b8f5

Comment 46

[Dzmitry Malyshau [:kvark]]

I also made a try push that force-disables the use of GL swizzling, to simulate the conditions we have on the machines where it's supported but black-listed. It works fine on my macbook Intel with Broadwell GPU. Let's see if there are any reports after https://phabricator.services.mozilla.com/D77896 is landed.

Comment 47

[Jeff Muizelaar [:jrmuizel]]

Robin, do you still see this problem?

Comment 48

[Jeff Muizelaar [:jrmuizel]]

Alan do you still see this problem?

Comment 49

[Robin Whittleton]

Not from me, I never saw it again after I verified fixed in https://bugzilla.mozilla.org/show_bug.cgi?id=1570736#c15

Comment 50

[Wayne Mery (:wsmwk)]

(ckoff and ajeffrey are gone)