1621523 - Firefox 74 has drawing issues with Amazon Sumerian Editor

Status: VERIFIED FIXED

(Core :: Graphics: CanvasWebGL, defect, P1)

Description

[Deirdre Toomey]

Attachment: drawing failure.png

User Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/80.0.3987.132 Safari/537.36

Steps to reproduce:

We updated to Firefox 74, and opened a scene in the Sumerian editor

Actual results:

Objects in the 3D Sumerian scene are not visible, only the grid. When viewed in play mode, with the grid disabled, we can see the objects rendered correctly. Based on Spector js captures, our conclusion is that the three transparent layers we have in the editor: gridRendering, cameraDebugRendering and LightDebugRendering, are not blending with underneath canvas the same way they did in previous versions. Instead, they are now overriding the canvas.

We are attempting to resolve the issue, but are reduced to diffing various builds until we find one that works. Do you all know of any changes to 3d blending or drawing schemas that may have changed in 74? Any help to resolve this issue would be appreciated.

I will update later with an observable scene, if possible

Expected results:

The 3D scene was previously drawing correctly.

Comment 1

[Deirdre Toomey]

Here is a Sumerian scene which is reproducing the issue. on FF74, the white box will not be visible

https://cff70084960a4674ba74c27903e361bc.us-east-2.sumerian.aws/

Comment 2

[Deirdre Toomey]

Attachment: an example of another level opened in FF74

an example of another level opened in FF74, to be compared with a pic of the same level loaded in chrome

Comment 3

[Deirdre Toomey]

Attachment: visible in Chrome81.png

Same level loaded in chrome

Comment 4

[Deirdre Toomey]

Attachment: hardware config

hardware and browser config from the machine loading the scene in FF74 and chrome

Comment 5

[Alice0775 White]

Regression window:
https://hg.mozilla.org/integration/autoland/pushloghtml?fromchange=04ada1c4f921220de2b708aef66748e1394354a2&tochange=2c23ef1249141c1cf9ca6e3043982f0e34b890eb

Comment 6

[Mike Taylor [:miketaylr]]

Jeff, can you take a look? thanks

Comment 7

[Jessie [:jbonisteel] pls NI]

Jeff is looking at this!

Comment 8

[Deirdre Toomey]

Hello!
If it's at all helpful, my devs have been doing a bunch of investigating, think the issue may have been introduced around Jan 15th, in this build: https://archive.mozilla.org/pub/firefox/nightly/2020/01/2020-01-15-09-38-07-mozilla-central/

Comment 9

[Kelsey Gilbert [:jgilbert]]

Comment #c5 pinpointed the regressing cset, but the bug in that change isn't obvious to me. I can reproduce it locally, and I'm able to reproduce it in a webgl record+replay. (replay even works in Chrome, same bug in Firefox) Next I'll be figuring out what's up by playing with the recording.

Comment 10

[Deirdre Toomey]

If there's anything we can provide to simplify your debugging and/or search, please let me know!

Comment 11

[Kelsey Gilbert [:jgilbert]]

Attachment: recording of the broken frame with bad{} section marked

Comment 12

[Kelsey Gilbert [:jgilbert]]

The main changed functionality that jumps out at me from comment 5 is that now every width/height assignment counts as an attempted Resize, at least on the Client side. I'm guessing that the issue is with colorMask/depthMask and WebGLContext::Resize.

Comment 13

[Kelsey Gilbert [:jgilbert]]

I bet it's depthMask, since the broken rendering we see in Firefox is only the grid-lines, which is a surprising partial rendering.

Comment 14

[Kelsey Gilbert [:jgilbert]]

Attachment: testcase-sumerian-mask.html

Chrome has a green dot, buggy Firefox has a blue dot.

Comment 15

[Kelsey Gilbert [:jgilbert]]

Attachment: testcase-sumerian-mask2.html

It looks like we're just clearing the depth buffer when we shouldn't.

Comment 16

[Kelsey Gilbert [:jgilbert]]

Attachment: testcase-sumerian-mask3.html

Resize() calls PresentScreenbuffer(), which with preserveDrawingBuffer:false, clears all the buffers, resetting both color and depth.

Comment 17

[Kelsey Gilbert [:jgilbert]]

The workaround is simple: Don't assign to canvas.width/height if they're already the values you want.

Comment 18

[Kelsey Gilbert [:jgilbert]]

Alternatively, you can use preserveDrawingBuffer:true for the time being.

Comment 19

[Jessie [:jbonisteel] pls NI]

Deirdre - jgilbert has two recommendations for potential workarounds in the two comments above. We will also uplift a patch on our end to Firefox Beta so this will be fixed in the next release.

Comment 20

[Deirdre Toomey]

Great, thank you very much! We are looking into implementing your suggested work-arounds, and will update here on how it goes!

Comment 21

[Deirdre Toomey]

Update: We were able to successfully work around the issue, and have released an update. Thank you all very much for your help!

Comment 22

[Kelsey Gilbert [:jgilbert]]

CTS test: https://github.com/KhronosGroup/WebGL/pull/3038

Comment 23

[Kelsey Gilbert [:jgilbert]]

Confirmed fix with the CTS test and the testcases HTMLs attached to this bug. URL link has been updated so it no longer repros on Firefox.

Comment 24

[Kelsey Gilbert [:jgilbert]]

Attachment: Bug 1621523 - WebGL: Don't Present on no-op Resize.

Comment 25

[Kelsey Gilbert [:jgilbert]]

Attachment: Bug 1621523 - Revendor WebGL CTS with new test.

Comment 26

[Deirdre Toomey]

One question for you all - this issue continues to impact scenes that were published in older versions of our tool... would it be possible to hotfix FF74, rather than having to ask all of our customers to republish if their scenes are experiencing difficulties in FF74?

Comment 27

[Kelsey Gilbert [:jgilbert]]

I don't think we can hotfix 74 at this point. Generally once changes as published to Release in Firefox, (as it is in Chrome) the bar for patching is very high.
Once it hits release, if it's not a security problem or something we can easily disable, it's generally WONTFIX.
To prevent this in the future, you really need to be testing in Firefox Nightly, or at least in Beta.

Comment 28

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/c997932b6bfe
WebGL: Don't Present on no-op Resize. r=lsalzman
https://hg.mozilla.org/integration/autoland/rev/12e09fef3c3f
Revendor WebGL CTS with new test.

Comment 29

[Deirdre Toomey]

Ok, understandable.

We unfortunately have hit a snag with the workaround - while it works with the test level we had provided, it does not ctually work with the Sumerian Editor. Can we set you up with the Editor to take another look?

Comment 30

[Razvan Maries]

Backed out for perma failures on test_2_conformance2__rendering__framebuffer-render-to-layer.html.

Push with failure: https://treeherder.mozilla.org/#/jobs?repo=autoland&selectedJob=293135529&resultStatus=testfailed%2Cbusted%2Cexception&revision=12e09fef3c3f9bebbe40bb018f1310616024588d

Failure log: https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=293135529&repo=autoland&lineNumber=4927

Backout: https://hg.mozilla.org/integration/autoland/rev/71ac44e7f4cdbf6c5eb85411b2bed2ced3b1c66b

Comment 31

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/a39217483d29
WebGL: Don't Present on no-op Resize. r=lsalzman
https://hg.mozilla.org/integration/autoland/rev/73572c522da8
Revendor WebGL CTS with new test.

Comment 32

[Narcis Beleuzu [:NarcisB]]

https://hg.mozilla.org/mozilla-central/rev/a39217483d29
https://hg.mozilla.org/mozilla-central/rev/73572c522da8

Comment 33

[Kelsey Gilbert [:jgilbert]]

Comment on attachment 9133292 [details]
Bug 1621523 - WebGL: Don't Present on no-op Resize.

Beta/Release Uplift Approval Request

• User impact if declined: Incomplete rendering in some content. (e.g. Amazon Sumerian editor)
• Is this code covered by automated tests?: Yes
• Has the fix been verified in Nightly?: No
• Needs manual test from QE?: Yes
• If yes, steps to reproduce: Testcases attached to this bug should look the same as they do in Chrome.
• List of other uplifts needed: None
• Risk to taking this patch: Low
• Why is the change risky/not risky? (and alternatives if risky): Low risk because the change is small, and uses existing machinery.
• String changes made/needed: none

Comment 34

[Kelsey Gilbert [:jgilbert]]

Comment on attachment 9133293 [details]
Bug 1621523 - Revendor WebGL CTS with new test.

Don't uplift the test updates.

Comment 35

[Kelsey Gilbert [:jgilbert]]

(In reply to Deirdre Toomey from comment #29)

Ok, understandable.

We unfortunately have hit a snag with the workaround - while it works with the test level we had provided, it does not ctually work with the Sumerian Editor. Can we set you up with the Editor to take another look?

Can you try replacing instances of canvas.width = x; with if (canvas.width != x) canvas.width = x;?

Comment 36

[Julien Cristau [:jcristau]]

Comment on attachment 9133292 [details]
Bug 1621523 - WebGL: Don't Present on no-op Resize.

webgl regression fix, approved for 75.0b5

Comment 37

[Razvan Maries]

https://hg.mozilla.org/releases/mozilla-beta/rev/bfa944211c91

Comment 38

[Deirdre Toomey]

Let me see...

Comment 39

[Deirdre Toomey]

wait - that's exactly what we did

Comment 40

[Deirdre Toomey]

it worked for the published scene, but not the Sumerian Editor

Comment 41

[Kelsey Gilbert [:jgilbert]]

(In reply to Deirdre Toomey from comment #40)

it worked for the published scene, but not the Sumerian Editor

Is the Editor fixed in Nightly now?

Comment 42

[Deirdre Toomey]

I'm confused - Was there a change checked in from your side?

Comment 43

[Kelsey Gilbert [:jgilbert]]

(In reply to Deirdre Toomey from comment #42)

I'm confused - Was there a change checked in from your side?

• An up-to-date Nightly should have the fix for all testcases in this bug
  • but I don't know if it fixed the Editor too
  • (The next Beta build we publish should also be fixed, but it might not be out yet)
• I think I saw that you published a workaround that fixed the published scenes in affected versions

If you still see a bug in the Editor in up-to-date Nightly, please file a new bug for it.

Comment 44

[Jessie [:jbonisteel] pls NI]

Deirdre - if you do still see the issue with your Editor, can you also send us info on how we can reproduce that on our end? (Not sure if we'd need access to the editor somehow)

Comment 45

[Rares Doghi, Desktop QA]

Using the test cases from Comment 15 and 16 I was able to confirm this issue Verified as Fixed on Windows 10 using Beta 75.0b5 and our latest Nightly build: 76.0a1 (2020-03-16).

I will update the flags for this issue and if the issue still occurs we can log a new issue for the Editor. I tried to test this issue on the Sumerian Editor but it needs a valid account.

Comment 46

[Deirdre Toomey]

Ok. Yes, this issue is fixed for the test scenes on FF74 we created to attempt to reproduce it, but are still happening in the editor on some machines, which requires a free AWS account to access. I will create a new ticket with instructions for how to best access and reproduce. Thanks!

We will also test against the nightly to see if it fixes the editor issues for FF75. (assuming my understanding here correct?)

Comment 47

[Kelsey Gilbert [:jgilbert]]

That's correct. The editor seems to work the same as Chrome now for me.

Comment 48

[Kelsey Gilbert [:jgilbert]]

(In reply to Deirdre Toomey from comment #46)

Ok. Yes, this issue is fixed for the test scenes on FF74 we created to attempt to reproduce it, but are still happening in the editor on some machines, which requires a free AWS account to access. I will create a new ticket with instructions for how to best access and reproduce. Thanks!

We will also test against the nightly to see if it fixes the editor issues for FF75. (assuming my understanding here correct?)

FWIW Nightly is now FF76, and Beta is 75.