1694824 - Enable Fission mochitest-browser-chrome tests on more platforms

Status: RESOLVED FIXED

(Testing :: Mochitest, task, P2)

Description

[Chris Peterson [:cpeterson]]

Our testing strategy and list of tests we'd like to enable on which platforms:

https://docs.google.com/document/d/19MergQfMob2mdb_s7hlz5Ilj7Bn-Bbi324r6d2bZfxw/edit

We want to run these tests on:

• debug and opt builds
  • Where “shippable” opt builds available, we prefer to test shippable opt builds on central and plain opt builds on autoland.
• autoland and central
• We prefer to run tests with Fission + WebRender (“qr”) where available.

Add platforms:

• linux1804-64-asan/opt
• linux1804-64-tsan/opt
• macosx1014-64-qr/debug
• macosx1014-64-qr/opt on autoland, shippable-qr/opt on central
• windows10-64-qr/debug (we’re already testing opt)

Also, mochitest-browser-chrome is running on Linux but without WebRender:

* test-linux1804-64-shippable/opt-mochitest-browser-chrome-fis-e10s
* test-linux1804-64/debug-mochitest-browser-chrome-fis-e10s

We would prefer to switch Linux to also use WebRender like:

* test-linux1804-64-qr/opt on autoland, shippable-qr/opt on central
* test-linux1804-64-qr/debug

(Edit: we decided that we won't enable WebRender for the Fission mochitest-browser-chrome tests at this time because of the test issues Andrew saw in comment 4.)

Remove platforms:

• windows10-64 opt fission without qr (because we already test windows10-64 opt fission+qr with WebRender)

Comment 1

[BugBot [:suhaib / :marco/ :calixte]]

The product::component has been changed since the backlog priority was decided, so we're resetting it.
For more information, please visit auto_nag documentation.

Comment 2

[Chris Peterson [:cpeterson]]

Assigning remaining "Enable Fission tests on more platforms" bugs to ahal.

Comment 3

[Andrew Halberstadt [:ahal]]

This bug has been added to releng's backlog and will be picked up sometime in the next few weeks. I'll likely end up taking this, but unassigning myself for now in case someone else ends up helping out.

Comment 4

[Andrew Halberstadt [:ahal]]

Hey Chris, a fair amount of failures here:
https://treeherder.mozilla.org/jobs?repo=try&group_state=expanded&revision=69b80166a97fbc51edfd3a4d60ec519a0b6359c5

On Linux, mochitest-browser-chrome was previously not running with webrender enabled at all, so it's not too surprising that there are more problems there. Maybe we should leave fission as non-webrender on Linux for now and work on enabling regular browser-chrome with webrender as a prerequisite..

I'll see if I can disable tests to victory on Mac and Windows.

Comment 5

[Chris Peterson [:cpeterson]]

(In reply to Andrew Halberstadt [:ahal] from comment #4)

On Linux, mochitest-browser-chrome was previously not running with webrender enabled at all, so it's not too surprising that there are more problems there. Maybe we should leave fission as non-webrender on Linux for now and work on enabling regular browser-chrome with webrender as a prerequisite..

That's a bummer. I'll follow up with the WebRender team about the failures.

In the meantime, let's follow your suggestion and leave Fission as non-WebRender for mochitest-browser-chrome on Linux. We'll probably have fewer problems, even though Fission without WebRender is not an officially-supported configuration.

I'll see if I can disable tests to victory on Mac and Windows.

If you will have to disable a lot of tests on Mac and Windows, feel free to instead try running mochitest-browser-chrome without WebRender there, too. Testing Fission, not WebRender, is the priority. :)

Edit: I see now that we already run mochitest-browser-chrome with WebRender for e10s on Mac and Windows, so any test problems with mochitest-browser-chrome for Fission+WebRender will be Fission's fault.

Comment 6

[Andrew Halberstadt [:ahal]]

Fission is running with webrender on Windows and Mac, it's just Linux that was missing for some reason. So issues there should be fission specific.

Though there were a few more failures there than I bargained for as well.

Comment 7

[Andrew Halberstadt [:ahal]]

Attachment: Bug 1694824 - [ci] Annotate browser-chrome manifests for new fission failures, r?jmaher!

Comment 8

[Andrew Halberstadt [:ahal]]

Attachment: Bug 1694824 - [ci] Enable more fission mochitest-browser-chrome tasks, r?jmaher!

This makes the following changes on autoland:
+test-linux1804-64/opt-mochitest-browser-chrome-fis-e10s
+test-macosx1015-64-qr/debug-mochitest-browser-chrome-fis-e10s
+test-macosx1015-64-qr/opt-mochitest-browser-chrome-fis-e10s
+test-windows10-64-qr/debug-mochitest-browser-chrome-fis-e10s
+test-windows10-64-qr/opt-mochitest-browser-chrome-fis-e10s

And the following changes on central:
+test-macosx1015-64-qr/debug-mochitest-browser-chrome-fis-e10s
+test-macosx1015-64-shippable-qr/opt-mochitest-browser-chrome-fis-e10s
+test-windows10-64-qr/debug-mochitest-browser-chrome-fis-e10s
-test-windows10-64-shippable/opt-mochitest-browser-chrome-fis-e10s

Depends on D116276

Comment 9

[Andrew Halberstadt [:ahal]]

Thanks for linking these, I'll adjust the script to add the link in the future.

Marking leave-open as I still haven't landed asan yet.

Comment 10

[Andrew Halberstadt [:ahal]]

Attachment: Bug 1694824 - [ci] Enable fission mochitest-browser-chrome with Linux asan, r?jmaher!

This makes the following changes on autoland:
+test-linux1804-64-asan/opt-mochitest-browser-chrome-fis-e10s

And the following changes on central:
+test-linux1804-64-asan/opt-mochitest-browser-chrome-fis-e10s

Depends on D116277

Comment 11

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/98a245b6834f
[ci] Annotate browser-chrome manifests for new fission failures, r=jmaher,extension-reviewers,preferences-reviewers,zombie
https://hg.mozilla.org/integration/autoland/rev/dc970a8384df
[ci] Enable more fission mochitest-browser-chrome tasks, r=jmaher
https://hg.mozilla.org/integration/autoland/rev/b432adba185f
[ci] Enable fission mochitest-browser-chrome with Linux asan, r=jmaher

Comment 12

[Sandor Molnar[:smolnar]]

https://hg.mozilla.org/mozilla-central/rev/98a245b6834f
https://hg.mozilla.org/mozilla-central/rev/dc970a8384df
https://hg.mozilla.org/mozilla-central/rev/b432adba185f

Comment 13

[Chris Peterson [:cpeterson]]

(In reply to Andrew Halberstadt [:ahal] from comment #9)

Marking leave-open as I still haven't landed asan yet.

Andrew, can we close this bug as fixed now? Looks like you landed Linux asan and it was successfully merged to mozilla-central:

(In reply to Pulsebot from comment #11)

https://hg.mozilla.org/integration/autoland/rev/b432adba185f
[ci] Enable fission mochitest-browser-chrome with Linux asan, r=jmaher

Comment 14

[Andrew Halberstadt [:ahal]]

Yes, thanks for noticing!