1694833 - Enable Fission mochitest-plain tests on more platforms

Status: RESOLVED FIXED

(Testing :: Mochitest, task, P2)

Description

[Chris Peterson [:cpeterson]]

Our testing strategy and list of tests we'd like to enable on which platforms:

https://docs.google.com/document/d/19MergQfMob2mdb_s7hlz5Ilj7Bn-Bbi324r6d2bZfxw/edit

We want to run these tests on:

• debug and opt builds
  • Where “shippable” opt builds available, we prefer to test shippable opt builds on central and plain opt builds on autoland.
• autoland and central
• We prefer to run tests with Fission + WebRender (“qr”) where available.

Add platforms:

• linux1804-64-asan/opt
• linux1804-64-tsan/opt
• macosx1014-64-qr/debug
• macosx1014-64-qr/opt on autoland, shippable-qr/opt on central
• windows10-64-qr/debug

Remove platforms:

• linux1804-64 opt fission without qr (because we already test linux1804-64 opt fission+qr)
• windows10-64 opt fission without qr (because we already test windows10-64 opt fission+qr)

Comment 1

[Neha Kochar [:neha]]

Andrew, can you please enable the additional platforms for these tests?

Comment 2

[Andrew Halberstadt [:ahal]]

This has been added to our current sprint.

Comment 3

[Andrew Halberstadt [:ahal]]

I wrote up a patch to add these changes, but looks like there are some failures to triage:
https://treeherder.mozilla.org/jobs?repo=try&author=ahalberstadt%40mozilla.com

(ignore the fission-xorigin tests, I added those to my try run by accident)

Looks like asan is pretty stable, I'll get that landed at least while the other failures are fixed or skipped.

Comment 4

[Andrew Halberstadt [:ahal]]

Attachment: Bug 1694833 - [ci] Move 'fission-run-on-projects' out of 'job-defaults' in mochitest.yml, r?jmaher

Only mochitest-plain and mochitest-media were inheriting this (the rest didn't
have fission in their variants or had an override for this key). Since
mochitest-plain is about to diverge later on this stack, moving it into the job
definition makes it easier to read.

Comment 5

[Andrew Halberstadt [:ahal]]

Attachment: Bug 1694833 - [ci] Remove non-qr mochitest-plain-fis tests from mozilla-central, r?jmaher

We don't need to run fission tests on both -qr and non-qr platforms with opt.
Remove the non-qr versions as we prefer running with webrender.

This patch removes the following tasks on mozilla-central:
-test-linux1804-64-shippable/opt-mochitest-plain-fis-e10s
-test-windows10-64-shippable/opt-mochitest-plain-fis-e10s

Depends on D108028

Comment 6

[Andrew Halberstadt [:ahal]]

Attachment: Bug 1694833 - [ci] Enable mochitest-plain-fis with linux64 asan, r?jmaher

Depends on D108029

Comment 7

[Chris Peterson [:cpeterson]]

(In reply to Andrew Halberstadt [:ahal] from comment #3)

I wrote up a patch to add these changes, but looks like there are some failures to triage:
https://treeherder.mozilla.org/jobs?repo=try&author=ahalberstadt%40mozilla.com

(ignore the fission-xorigin tests, I added those to my try run by accident)

Just let me know when you'd like me to ask a Fission engineer to triage those mochitest-plain-fis failures.

https://treeherder.mozilla.org/jobs?repo=try&revision=8c314dcd7d37e0e83d8248c6404df50e36d4768c

Comment 8

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/94cfc391dd5e
[ci] Move 'fission-run-on-projects' out of 'job-defaults' in mochitest.yml, r=jmaher
https://hg.mozilla.org/integration/autoland/rev/8e3e01b3a52e
[ci] Remove non-qr mochitest-plain-fis tests from mozilla-central, r=jmaher
https://hg.mozilla.org/integration/autoland/rev/df572675f5ee
[ci] Enable mochitest-plain-fis with linux64 asan, r=jmaher

Comment 9

[Andrew Halberstadt [:ahal]]

Will do! Was just consulting with jmaher on this. I'll try to get it green via test disabling for now so we can get it landed sooner.

Comment 10

[Cristian Brindusan [:cbrindusan]]

https://hg.mozilla.org/mozilla-central/rev/94cfc391dd5e
https://hg.mozilla.org/mozilla-central/rev/8e3e01b3a52e
https://hg.mozilla.org/mozilla-central/rev/df572675f5ee

Comment 11

[Andrew Halberstadt [:ahal]]

Attachment: Bug 1694833 - Skip some mochitest-plain tests with fission so we can add new tasks, r?jmaher

Comment 12

[Andrew Halberstadt [:ahal]]

Attachment: Bug 1694833 - [ci] Enable mochitest-plain-fis on more platforms, r?jmaher

This patch enables the following tasks on mozilla-central:

test-macosx1014-64-qr/debug-mochitest-plain-fis-e10s
test-macosx1014-64-shippable-qr/opt-mochitest-plain-fis-e10s
test-windows10-64-qr/debug-mochitest-plain-fis-e10s

And the following tasks on autoland:

test-linux1804-64-qr/opt-mochitest-plain-fis-e10s
test-macosx1014-64-qr/debug-mochitest-plain-fis-e10s
test-macosx1014-64-qr/opt-mochitest-plain-fis-e10s
test-windows10-64-qr/debug-mochitest-plain-fis-e10s
test-windows10-64-qr/opt-mochitest-plain-fis-e10s

Depends on D108251

Comment 13

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/4b99cb914aad
Skip some mochitest-plain tests with fission so we can add new tasks, r=jmaher
https://hg.mozilla.org/integration/autoland/rev/726db4506c75
[ci] Enable mochitest-plain-fis on more platforms, r=jmaher

Comment 14

[Andrew Halberstadt [:ahal]]

Hey Chris, needinfo just as an fyi.

In the process of landing the above I disabled the following two tests on Windows and Mac:

layout/base/tests/test_bug851485.html
toolkit/components/url-classifier/tests/mochitest/test_classifier.html

Both had frequent intermittents.

(In reply to Chris Peterson [:cpeterson] from comment #0)

Add platforms:

• linux1804-64-tsan/opt

Additionally tsan doesn't quite seem ready as there a bunch of failures that don't seem related to tests. See:
https://treeherder.mozilla.org/jobs?repo=try&revision=c50dd851c95cc3b8d0cd1e6cd4889b6a6a4d842a&searchStr=tsan&group_state=expanded

According to Push Health, there are 7 possible regressions in there (though likely fewer than that as some look to be the same issue).

Just to be clear, I consider this bug done on my part until I hear otherwise.

Comment 15

[Chris Peterson [:cpeterson]]

(In reply to Andrew Halberstadt [:ahal] from comment #14)

In the process of landing the above I disabled the following two tests on Windows and Mac:

layout/base/tests/test_bug851485.html
toolkit/components/url-classifier/tests/mochitest/test_classifier.html

Both had frequent intermittents.

Thanks. I will file new bugs for the Fission team to investigate: bug 1678089 and bug 1698195.

Additionally tsan doesn't quite seem ready as there a bunch of failures that don't seem related to tests. See:
https://treeherder.mozilla.org/jobs?repo=try&revision=c50dd851c95cc3b8d0cd1e6cd4889b6a6a4d842a&searchStr=tsan&group_state=expanded

According to Push Health, there are 7 possible regressions in there (though likely fewer than that as some look to be the same issue).

That's unfortunate. mochitest-plain already runs on linux1804-64-tsan/opt for e10s (non-Fission), so Fission must causing these new failures. I will file a new bug for the Fission team to investigate: bug 1698193.

Just to be clear, I consider this bug done on my part until I hear otherwise.

Yes. Thanks! I'll remove this bug's leave-open tag.

Comment 16

[Narcis Beleuzu [:NarcisB]]

https://hg.mozilla.org/mozilla-central/rev/4b99cb914aad
https://hg.mozilla.org/mozilla-central/rev/726db4506c75