1673071 - Flip macOS tests to running mostly with WR-enabled

Status: RESOLVED FIXED

(Core :: Graphics: WebRender, task)

Description

[Kartikaya Gupta (email:[e-mail])]

WebRender is getting turned on for 90%+ of the macOS population in Fx 83. So we should ensure that most of the CI jobs we run for macOS are doing so with WR enabled. We should still leave some sanity tests for the layers codepath.

Comment 1

[Kartikaya Gupta (email:[e-mail])]

https://treeherder.mozilla.org/#/jobs?repo=try&group_state=expanded&revision=71885dcb6cd8603001ac8367d6ecdabee2de0171

This is a try push where I simply copied all the suites that run for macOS (layers) and put them under macOS (QR). Seems mostly in good shape but a bunch of tests might need disabling and/or fuzzing or expectation fixing.

Comment 2

[Kartikaya Gupta (email:[e-mail])]

https://treeherder.mozilla.org/#/jobs?repo=try&group_state=expanded&revision=11ca8e035f57725a0fc142ce856bd7bd0cc5df44 should have all the right taskcluster config changes. It basically flips around the current macOS coverage so that all the jobs that were running without WR before are now running with WR. And the jobs that were running with WR are now running on macosx1014-64/debug and macosx1014-64-devedition/opt. I also adjusted some of the test expectations from the last try push but there will be some more tests that I need to skip with WR enabled (at least those covered by bug 1673105, and maybe a couple more).

Comment 3

[Kartikaya Gupta (email:[e-mail])]

There are two failures which are due to MOZ_ASSERT failures, but I can't reproduce those locally. Bug 1674411 and bug 1674413 track those and I'm just going to mark those tests as skip-if for now. Bug 1674412 seems intermittent so for now I'll leave it as running and if the frequency of failure is too high we can try to do something about it. Bug 1674060 was filed for failures on windows-ccov-qr but I'm seeing those on macosx-qr as well, so I'm going to skip the two tests involved in that as well.

Comment 4

[Kartikaya Gupta (email:[e-mail])]

https://treeherder.mozilla.org/jobs?repo=try&group_state=expanded&revision=f39b076429f66e6de4b24fa4b8ef262821c2cac8 was my last try push. The marionette mozinfo object appears to not have the right value for the webrender field, so my skip-if didn't take effect. And I missed some of the webgl failures since the TreeHerder summary didn't show them all the last time around. Both of those problems should be fixed in this new push:

https://treeherder.mozilla.org/jobs?repo=try&group_state=expanded&revision=f07fa06a4de865fe1a611ed720a1c4cf55d2badd

Comment 5

[Kartikaya Gupta (email:[e-mail])]

Attachment: Bug 1673071 - Update some test expectations for macOS+WR configuration. r?jrmuizel

Comment 6

[Kartikaya Gupta (email:[e-mail])]

Attachment: Bug 1673071 - Update marionette mozinfo object with correct webrender status. r?whimboo

Depends on D95427

Comment 7

[Kartikaya Gupta (email:[e-mail])]

Attachment: Bug 1673071 - Disable failing tests. r?jrmuizel

Depends on D95428

Comment 8

[Kartikaya Gupta (email:[e-mail])]

Attachment: Bug 1673071 - Run the full slate of test suites on macos-qr. r?jmaher!,jrmuizel!

This patch flips things around so that the non-QR macosx test platforms are
running a "skeleton crew" set of tests, and the QR macosx test platforms are
running all the tests that were previously running with non-QR. As of Fx83
macOS has WebRender enabled by default for 90%+ of the population so it makes
sense to test that configuration more heavily, but leave some tests running
for the non-QR configuration as well.

Depends on D95429

Comment 9

[Kartikaya Gupta (email:[e-mail])]

With comments addressed and rebased: https://treeherder.mozilla.org/jobs?repo=try&revision=7303df5fd181ad5e02a06520507cd16391f7c841

Comment 10

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/d6857ecf3c81
Update some test expectations for macOS+WR configuration. r=jrmuizel
https://hg.mozilla.org/integration/autoland/rev/34d6c0fbfeef
Update marionette mozinfo object with correct webrender status. r=whimboo,marionette-reviewers
https://hg.mozilla.org/integration/autoland/rev/f2ed8ed80bf2
Disable failing tests. r=jrmuizel,marionette-reviewers,jgilbert
https://hg.mozilla.org/integration/autoland/rev/a8ad01797ee0
Run the full slate of test suites on macos-qr. r=jmaher,jrmuizel

Comment 11

[Razvan Maries]

Backed out for perma failures.

Push with failures: https://treeherder.mozilla.org/jobs?repo=autoland&selectedTaskRun=IPeLCgoiShiz9hqMNXRiNw.0&resultStatus=pending%2Crunning%2Csuccess%2Ctestfailed%2Cbusted%2Cexception&searchStr=android%2C7.0%2Cx86-64%2Cdebug%2Cweb%2Cplatform%2Ctests%2Ctest-android-em-7.0-x86_64%2Fdebug-geckoview-web-platform-tests-e10s%2Cwpt16&revision=a8ad01797ee0d2ee1c068945428fd57d9b2c0a4e

Log: https://treeherder.mozilla.org/logviewer?job_id=320649027&repo=autoland&lineNumber=2848

Backout: https://hg.mozilla.org/integration/autoland/rev/05a9140d0dc9809c086699ac55ff9e4612d9faa1

Comment 12

[Kartikaya Gupta (email:[e-mail])]

Ah, stupid mistake in my updating of the test expectations.

Comment 13

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/5af0cc839a14
Update some test expectations for macOS+WR configuration. r=jrmuizel
https://hg.mozilla.org/integration/autoland/rev/2c4c6edf9856
Update marionette mozinfo object with correct webrender status. r=whimboo,marionette-reviewers
https://hg.mozilla.org/integration/autoland/rev/129e006c7b76
Disable failing tests. r=jrmuizel,marionette-reviewers,jgilbert
https://hg.mozilla.org/integration/autoland/rev/aabd0cfc1ac7
Run the full slate of test suites on macos-qr. r=jmaher,jrmuizel

Comment 14

[Cristian Brindusan [:cbrindusan]]

https://hg.mozilla.org/mozilla-central/rev/5af0cc839a14
https://hg.mozilla.org/mozilla-central/rev/2c4c6edf9856
https://hg.mozilla.org/mozilla-central/rev/129e006c7b76
https://hg.mozilla.org/mozilla-central/rev/aabd0cfc1ac7