1919114 - (gpu-process-testing) Add testing variant for disabling GPU process

Status: RESOLVED FIXED

(Core :: Graphics, task, P3)

Description

[Brad Werth [:bradwerth]]

An isolated GPU process is currently on-by-default for Windows and Android, and available but off-by-default for Linux X11. As we expand this capability to more platforms, and change the feature to on-by-default on more platforms, it will be useful to have a test variant that can compare test behavior with or without the isolated GPU process. Since the goal is to have GPU process on-by-default for all platforms, the variant should probably be a "no-gpu-process" variant that will eventually be removed once the feature is mature.

Comment 1

[Brad Werth [:bradwerth]]

Attachment: Bug 1919114 Part 2: Add a no-gpu-process test variant for reftests and media mochitests.

This allows us to test platforms that support GPU process without
enabling the GPU process. It may help us flush out differing test
outcomes between the two cases. Eventually this will be retired when all
platforms have mature GPU process implementations.

Comment 2

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/c61e536db0ba
Add a no-gpu-process test variant for reftests and media mochitests. r=taskgraph-reviewers,gfx-reviewers,jcristau,jmaher,aosmond

Comment 3

[Sandor Molnar[:smolnar]]

Backed out for causing mochitest mda perma failures + reftest failures

Backout link: https://hg.mozilla.org/integration/autoland/rev/105391056fe502d3a27b698eb0239b85693b334e

Push with failures

Failure log mda -> Assertion failure: false (Failed to create video bridge), at /builds/worker/checkouts/gecko/ipc/glue/UtilityProcessManager.cpp

Failure log reftest ->REFTEST TEST-UNEXPECTED-FAIL | layout/reftests/bugs/289480.html#top == layout/reftests/bugs/289480-ref.html

Comment 4

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/799295025870
Add a no-gpu-process test variant for reftests and media mochitests. r=taskgraph-reviewers,gfx-reviewers,jcristau,jmaher,aosmond

Comment 5

[Atila Butkovits]

Backed out for causing failures at UtilityProcessManager.cpp.

Backout link: https://hg.mozilla.org/integration/autoland/rev/28cd4bcb06170c48fb041cbd4a7fc5a50897e5ef

Push where failures started: https://treeherder.mozilla.org/jobs?repo=autoland&selectedTaskRun=a0xM66GuQvuRjJcMNSXmhg.0&resultStatus=testfailed%2Cbusted%2Cexception&revision=1dc5ccccdeae3b472fd5ef136e875fbebd5ab046

Failure log: https://treeherder.mozilla.org/logviewer?job_id=475039852&repo=autoland&lineNumber=2191

Comment 6

[Brad Werth [:bradwerth]]

Attachment: Bug 1919114 Part 1: Allow UtilityAudioDecoderChild::CreateVideoBridge to work without a GPU process.

At some point, UtilityAudioDecoderChild lost the ability to work
without a GPU process. This patch remedies that by creating an
alternate connection for the endpoint, modeled after the code in
RDDProcessManager.

Comment 7

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/4054f51f1754
Part 1: Allow UtilityAudioDecoderChild::CreateVideoBridge to work without a GPU process. r=aosmond
https://hg.mozilla.org/integration/autoland/rev/87a86132adad
Part 2: Add a no-gpu-process test variant for reftests and media mochitests. r=taskgraph-reviewers,gfx-reviewers,jcristau,jmaher,aosmond

Comment 8

[Norisz Fay [:noriszfay]]

Backed out for causing multiple R-nogpu failures

Backout link

Push with failures

The failures can be seen in the above link. The tier 2 ones seem to be failing the same on each push.

Comment 9

[Brad Werth [:bradwerth]]

Attachment: Bug 1919114 Part 3: Update fuzziness of reftests with no gpu process.

Allowing (once again) a no-gpu-process option increases the number of
pixels which get antialiased differently than the reference image. The
images continue to largely match; it's just an increase in the number of
close-but-not-quite pixels.

Comment 10

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/bd75acb8f7ff
Part 1: Allow UtilityAudioDecoderChild::CreateVideoBridge to work without a GPU process. r=aosmond
https://hg.mozilla.org/integration/autoland/rev/e4cf55f1c4da
Part 2: Add a no-gpu-process test variant for reftests and media mochitests. r=taskgraph-reviewers,gfx-reviewers,jcristau,jmaher,aosmond
https://hg.mozilla.org/integration/autoland/rev/a69fadab030b
Part 3: Update fuzziness of Android tests with no gpu process. r=gfx-reviewers,aosmond

Comment 11

[Iulian Moraru]

Backed out for causing multiple reftest failures on multiple platforms with no-gpu enabled flavors.

• Push with failures

To check all the failures caused by this, please visit this link -> it has a filter that shows all the failures marked as fix by commit with this backout.

• Backout link
  This also caused Bug 1920837 and Bug 1920839, this bugs are closed after the backout.

Comment 12

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/3b459fd776e6
Part 1: Allow UtilityAudioDecoderChild::CreateVideoBridge to work without a GPU process. r=aosmond
https://hg.mozilla.org/integration/autoland/rev/9db329e6c216
Part 2: Add a no-gpu-process test variant for reftests and media mochitests. r=taskgraph-reviewers,gfx-reviewers,jcristau,jmaher,aosmond
https://hg.mozilla.org/integration/autoland/rev/0852cb6ea78f
Part 3: Update fuzziness of reftests with no gpu process. r=gfx-reviewers,aosmond

Comment 13

[Atila Butkovits]

Backed out for causing refetest failures.

Backout link: https://hg.mozilla.org/integration/autoland/rev/9ad4972f496a5553570d2e665a33d75e17ba44ff

Failures started here: https://treeherder.mozilla.org/jobs?repo=autoland&resultStatus=testfailed%2Cbusted%2Cexception%2Cretry%2Cusercancel&searchStr=reftest&revision=ea17a1d18e7dec25c08dd1eba7fb3b985fc0861e

Failure log: https://treeherder.mozilla.org/logviewer?job_id=475839772&repo=autoland&lineNumber=14617

Comment 14

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/ea1c122a3848
Part 1: Allow UtilityAudioDecoderChild::CreateVideoBridge to work without a GPU process. r=aosmond
https://hg.mozilla.org/integration/autoland/rev/226668ab3f35
Part 2: Add a no-gpu-process test variant for reftests and media mochitests. r=taskgraph-reviewers,gfx-reviewers,jcristau,jmaher,aosmond
https://hg.mozilla.org/integration/autoland/rev/bd7b34ae33ef
Part 3: Update fuzziness of reftests with no gpu process. r=gfx-reviewers,aosmond,taskgraph-reviewers,jmaher

Comment 15

[Atila Butkovits]

https://hg.mozilla.org/mozilla-central/rev/ea1c122a3848
https://hg.mozilla.org/mozilla-central/rev/226668ab3f35
https://hg.mozilla.org/mozilla-central/rev/bd7b34ae33ef

Comment 16

[Brad Werth [:bradwerth]]

Hopefully this one will stick, thank you all for your patience on this.

Comment 17

[Iulian Moraru]

Backed out for causing multiple nogpu reftest failures.

• Push that got backed out

You can check the failures in this link

• Backout link

Comment 18

[Iulian Moraru]

Backout merged to central: https://hg.mozilla.org/mozilla-central/rev/9354af15899d

Comment 19

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/cec6c1dcdfae
Part 1: Allow UtilityAudioDecoderChild::CreateVideoBridge to work without a GPU process. r=aosmond
https://hg.mozilla.org/integration/autoland/rev/f8e4f207430b
Part 2: Add a no-gpu-process test variant for reftests and media mochitests. r=taskgraph-reviewers,gfx-reviewers,jcristau,jmaher,aosmond
https://hg.mozilla.org/integration/autoland/rev/e7b92733fa76
Part 3: Update fuzziness of reftests with no gpu process. r=gfx-reviewers,aosmond,taskgraph-reviewers,jmaher

Comment 20

[Serban Stanca [:SerbanS]]

Backed out for causing no-gpu related reftests failures.

• Backout link
• Push with failures
• Failure Log
• Failure line: REFTEST TEST-UNEXPECTED-PASS | layout/reftests/bugs/1412375.html == layout/reftests/bugs/1412375-ref.html | image comparison, max difference: 0, number of differing pixels: 0

Comment 21

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/eeba73c31be2
Part 1: Allow UtilityAudioDecoderChild::CreateVideoBridge to work without a GPU process. r=aosmond
https://hg.mozilla.org/integration/autoland/rev/1a78a03693b0
Part 2: Add a no-gpu-process test variant for reftests and media mochitests. r=taskgraph-reviewers,gfx-reviewers,jcristau,jmaher,aosmond
https://hg.mozilla.org/integration/autoland/rev/d64a523fd44c
Part 3: Update fuzziness of reftests with no gpu process. r=gfx-reviewers,aosmond,taskgraph-reviewers,jmaher

Comment 22

[Iulian Moraru]

Backed out for causing multiple no-gpu reftest failures.

• Push with failures

Same issues as the backouts that I've made before.

• Backout link

Comment 23

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/9c9e6df18d35
Part 1: Allow UtilityAudioDecoderChild::CreateVideoBridge to work without a GPU process. r=aosmond
https://hg.mozilla.org/integration/autoland/rev/9fbb4bf4480d
Part 2: Add a no-gpu-process test variant for reftests and media mochitests. r=taskgraph-reviewers,gfx-reviewers,jcristau,jmaher,aosmond
https://hg.mozilla.org/integration/autoland/rev/9e75159e3f24
Part 3: Update fuzziness of reftests with no gpu process. r=gfx-reviewers,aosmond,taskgraph-reviewers,jmaher

Comment 24

[Serban Stanca [:SerbanS]]

Backed out for causing nogpu related reftests failures.

• Backout link
• Push with failures
• Failure Log
• Failure line: REFTEST TEST-UNEXPECTED-FAIL | layout/reftests/bugs/1840511-2.html == layout/reftests/bugs/1840511-1-ref.html | image comparison, max difference: 1, number of differing pixels: 309

Comment 25

[Brad Werth [:bradwerth]]

Apologies to the sheriffs and to anybody else watching this Bug or waiting for it to land. The failures are already filed as intermittents, but it's not acceptable to turn them into perma-fails by landing this Bug. I should have understood that sooner; I'm sorry! There's a couple of things I'll have to do differently to make this landable. Here's what I've realized about the test failures that keep bouncing this Bug:

1. Some tests already in the tree behave very differently with and without a gpu process. Since some platforms have one and some don't, this is expressed as very complex fuzzy-if declarations. A declaration like this is trying to tease out whether or not a gpu process is in use. The changes in this Bug are altering those conditions, which is leading to failures in tests.
2. The intermittent Bugs already filed on some of the affected tests are needed because sometimes these tests downgrade to abandon the gpu process after enough crashes in the process. So if a test is run after the gpu process is abandoned, it'll have a very different outcome. Bug 1895731 is an example. Without a gpu process, it draws nothing at all. This sometimes happen even on platforms with a gpu process, thus the intermittent.

So what to do about this? The ideal would be to update the sandbox with a variable indicating whether or not there is a gpu process in use. Unfortunately, the presence of the gpu process is not exposed to JS and I'm not sure how to do that. An alternative would be to add a variable that indicates whether a gpu process is possible (based on prefs and our current platform support) which will be mostly the same. Such a variable would be useful in simplifying the fuzzy-if logic for Case 1, above. It won't help with the Case 2 intermittents.

So that's what I'm going to do. I'll add nogpu as a variable to the sandbox and fuzz or disable tests that behave very differently with that variable set to true. It will be temporarily annoying to have a variable expressing the absence of something, but as every platform is expected to support a gpu process in time, then this variable and all the annotations around it will eventually be removed.

Comment 26

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/5c1e3623cdbd
Part 1: Allow UtilityAudioDecoderChild::CreateVideoBridge to work without a GPU process. r=aosmond
https://hg.mozilla.org/integration/autoland/rev/0177073b7b42
Part 2: Add a no-gpu-process test variant for reftests and media mochitests. r=taskgraph-reviewers,gfx-reviewers,jcristau,jmaher,aosmond
https://hg.mozilla.org/integration/autoland/rev/994aece36504
Part 3: Update fuzziness of reftests with no gpu process. r=gfx-reviewers,aosmond,taskgraph-reviewers,jmaher

Comment 27

[tszentpeteri]

Backed out for causing reftest failures on background-blending-color.html

• Backout link
• Push with failures
• Failure Log 1:
• Failure Log 2:

[task 2024-10-08T17:54:45.091Z] 17:54:45     INFO -  REFTEST TEST-START | layout/reftests/css-blending/background-blending-color.html == layout/reftests/css-blending/background-blending-color-ref.svg
[task 2024-10-08T17:54:45.091Z] 17:54:45     INFO -  REFTEST TEST-LOAD | http://10.0.2.2:8854/tests/layout/reftests/css-blending/background-blending-color.html | 61 / 1022 (5%)
[task 2024-10-08T17:54:45.091Z] 17:54:45     INFO -  REFTEST TEST-LOAD | http://10.0.2.2:8854/tests/layout/reftests/css-blending/background-blending-color-ref.svg | 61 / 1022 (5%)
[task 2024-10-08T17:54:45.091Z] 17:54:45     INFO -  REFTEST INFO | REFTEST wr-on-android dropping fuzz of (1, 9600) to (0, 0)
[task 2024-10-08T17:54:45.092Z] 17:54:45     INFO -  REFTEST INFO | REFTEST fuzzy test (1, 8000) <= (0, 0) <= (1, 9600)
[task 2024-10-08T17:54:45.092Z] 17:54:45  WARNING -  REFTEST TEST-UNEXPECTED-PASS | layout/reftests/css-blending/background-blending-color.html == layout/reftests/css-blending/background-blending-color-ref.svg | image comparison, max difference: 0, number of differing pixels: 0
<...>
[task 2024-10-08T17:54:45.135Z] 17:54:45     INFO -  REFTEST TEST-END | layout/reftests/css-blending/background-blending-color.html == layout/reftests/css-blending/background-blending-color-ref.svg

Comment 28

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/d4ab64473597
Part 1: Allow UtilityAudioDecoderChild::CreateVideoBridge to work without a GPU process. r=aosmond
https://hg.mozilla.org/integration/autoland/rev/46baa027678a
Part 2: Add a no-gpu-process test variant for reftests and media mochitests. r=taskgraph-reviewers,gfx-reviewers,jcristau,jmaher,aosmond
https://hg.mozilla.org/integration/autoland/rev/ed35ad3fdf9c
Part 3: Update fuzziness of reftests with no gpu process. r=gfx-reviewers,aosmond,taskgraph-reviewers,jmaher

Comment 29

[Pulsebot]

Backout by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/8a1fa23df5f9
Backed out 3 changesets for causing reftest failures on 1412375. CLOSED TREE

Comment 30

[Cosmin Sabou [:CosminS]]

Backed out for causing reftest failures on 1412375.

Push with failures: https://treeherder.mozilla.org/jobs?repo=autoland&resultStatus=testfailed%2Cbusted%2Cexception%2Cretry%2Cusercancel&revision=ed35ad3fdf9ca19da58b40a5a8e494b6c6115e44&selectedTaskRun=KuFdpk_eQY-sz_UtosgkhQ.0

Failure log: https://treeherder.mozilla.org/logviewer?job_id=477481117&repo=autoland

Backout link: https://hg.mozilla.org/integration/autoland/rev/8a1fa23df5f998a79ab500c157ed3150ad1b163e

Comment 31

[Timothy Nikkel (:tnikkel)]

fuzzy-if((!nogpu&&....

on the failing test. Did you mean to have the ! in there?

Comment 32

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/90218d9fb410
Part 1: Allow UtilityAudioDecoderChild::CreateVideoBridge to work without a GPU process. r=aosmond
https://hg.mozilla.org/integration/autoland/rev/c8be4eae972c
Part 2: Add a no-gpu-process test variant for reftests and media mochitests. r=taskgraph-reviewers,gfx-reviewers,jcristau,jmaher,aosmond
https://hg.mozilla.org/integration/autoland/rev/5c17ec5dca7b
Part 3: Update fuzziness of reftests with no gpu process. r=gfx-reviewers,aosmond,taskgraph-reviewers,jmaher

Comment 33

[Cristian Tuns]

https://hg.mozilla.org/mozilla-central/rev/90218d9fb410
https://hg.mozilla.org/mozilla-central/rev/c8be4eae972c
https://hg.mozilla.org/mozilla-central/rev/5c17ec5dca7b

Comment 34

[Brad Werth [:bradwerth]]

Looks like this will finally stick.

Comment 35

[Timothy Nikkel (:tnikkel)]

Attachment: Bug 1919114. Extend expiration of `no-gpu-process` variant. r?#gfx-reviewers

Comment 36

[Phabricator Automation]

Comment on attachment 9466114 [details]
Bug 1919114. Extend expiration of no-gpu-process variant. r?#gfx-reviewers

Revision D238163 was moved to bug 1948189. Setting attachment 9466114 [details] to obsolete.