1696042 - Enable Fission web-platform-tests on more platforms (Linux ASan and Windows 10 debug)

Status: RESOLVED FIXED

(Testing :: web-platform-tests, task, P2)

Description

[Chris Peterson [:cpeterson]]

Our testing strategy and list of tests we'd like to enable on which platforms:

https://docs.google.com/document/d/19MergQfMob2mdb_s7hlz5Ilj7Bn-Bbi324r6d2bZfxw/edit

We want to run these tests on:

• debug and opt builds
  • Where “shippable” opt builds available, we prefer to test shippable opt builds on central and plain opt builds on autoland.
• autoland and central
• We prefer to run tests with Fission + WebRender (“qr”) where available.

ASan and TSan are not run on all platforms.

We want to enable these WPTs on the following platforms:

• web-platform-tests
  • linux1804-64-asan/opt
  • linux1804-64-tsan/opt
  • windows10-64-qr/debug (we’re already testing opt)
• web-platform-tests-crashtest
  • linux1804-64-asan/opt
  • windows10-64-qr/debug (we’re already testing opt)
• web-platform-tests-print-reftest
  • linux1804-64-asan/opt
  • windows10-64-qr/debug (we’re already testing opt)
• web-platform-tests-reftest
  • linux1804-64-asan/opt
  • linux1804-64-tsan/opt
  • windows10-64-qr/debug (we’re already testing opt)
• web-platform-tests-wdspec
  • linux1804-64-asan/opt
  • windows10-64-qr/debug (we’re already testing opt)
• web-platform-tests-wdspec-headless
  • linux1804-64-qr/opt on autoland, shippable-qr opt on central
  • windows10-64-qr/opt on autoland, shippable-qr opt on central

Comment 1

[Neha Kochar [:neha]]

James, can you please enable the additional platforms for these WPTs?

Comment 2

[James Graham [:jgraham]]

https://treeherder.mozilla.org/jobs?repo=try&tier=1%2C2%2C3&revision=314f029c210e2eec270589b323424c4c24542742 is a try push with some extra platforms. Compared to the above list:

• tsan seems too unstable (in general, not fission specific) to enable more platforms at this time
• I can't quite tell what's going on with headless mode; there seems to be some special magic in the task configuration that prevents it being scheduled. I'll investigate further.

We can obviously update the metadata for the failures in the above try push.

Comment 3

[Neha Kochar [:neha]]

James, any more info from your investigation?

Comment 4

[Chris Peterson [:cpeterson]]

Assigning remaining "Enable Fission tests on more platforms" bugs to ahal.

Comment 5

[Andrew Halberstadt [:ahal]]

This bug has been added to releng's backlog and will be picked up sometime in the next few weeks. I'll likely end up taking this, but unassigning myself for now in case someone else ends up helping out.

Comment 6

[Andrew Halberstadt [:ahal]]

Attachment: Bug 1696042 - [ci] web-platform-test: Run fission debug on autoland and mozilla-central, r?jgraham

This adds the following configurations on central and autoland:

test-windows10-64-qr/debug-web-platform-tests-fis-e10s

Comment 7

[Andrew Halberstadt [:ahal]]

Attachment: Bug 1696042 - [ci] web-platform-test: Run fission linux64-asan/opt on autoland and mozilla-central

This adds the following configurations on central and autoland:

test-linux1804-64-asan/opt-web-platform-tests-fis-e10s

Depends on D110553

Comment 8

[Andrew Halberstadt [:ahal]]

Attachment: WIP: Bug 1696042 - [ci] web-platform-test: Run fission linux64-tsan on trunk

This enables the following configuration on central / autoland:

test-linux1804-64-tsan/opt-web-platform-tests-fis-e10s

Depends on D110554

Comment 9

[Andrew Halberstadt [:ahal]]

Attachment: Bug 1696042 - [ci] web-platform-test-crashtest: Run fission windows10-qr/debug on trunk, r?jgraham

Depends on D110555

Comment 10

[Andrew Halberstadt [:ahal]]

Attachment: Bug 1696042 - [ci] web-platform-test-crashtest: Run fission linux64-asan/opt on trunk, r?jgraham

This enables the following configuration on autoland/central:

test-linux1804-64-asan/opt-web-platform-tests-crashtest-fis-e10s

Comment 11

[Andrew Halberstadt [:ahal]]

Attachment: Bug 1696042 - [ci] web-platform-test-reftest: Run fission windows10-qr/debug on trunk, r?jgraham

This enables the following configuration on autoland/central:

test-windows10-64-qr/debug-web-platform-tests-reftest-fis-e10s

Comment 12

[Andrew Halberstadt [:ahal]]

Attachment: Bug 1696042 - [ci] web-platform-test-reftest: Run fission linux64-asan/opt on trunk, r?jgraham

This enables the following configuration on autoland/central:

test-linux1804-64-asan/opt-web-platform-tests-reftest-fis-e10s

Depends on D110830

Comment 13

[Andrew Halberstadt [:ahal]]

Attachment: Bug 1696042 - [ci] web-platform-test-print-reftest: Run fission windows10-qr/debug on trunk, r?jgraham

This enables the following configuration on autoland/central:

test-windows10-64-qr/debug-web-platform-tests-print-reftest-fis-e10s

Comment 14

[Andrew Halberstadt [:ahal]]

Attachment: Bug 1696042 - [ci] web-platform-test-print-reftest: Run fission linux64-asan/opt on trunk, r?jgraham

This enables the following configuration on autoland/central:

test-linux1804-64-asan/opt-web-platform-tests-print-reftest-fis-e10s

Depends on D110836

Comment 15

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/80e6aef40e50
[ci] web-platform-test: Run fission debug on autoland and mozilla-central, r=jgraham
https://hg.mozilla.org/integration/autoland/rev/5991f2ed9496
[ci] web-platform-test: Run fission linux64-asan/opt on autoland and mozilla-central r=jgraham
https://hg.mozilla.org/integration/autoland/rev/5802c0beafd1
[ci] web-platform-test-crashtest: Run fission windows10-qr/debug on trunk, r=jgraham
https://hg.mozilla.org/integration/autoland/rev/a62f976a393a
[ci] web-platform-test-crashtest: Run fission linux64-asan/opt on trunk, r=jgraham
https://hg.mozilla.org/integration/autoland/rev/d9de90fb0b90
[ci] web-platform-test-reftest: Run fission windows10-qr/debug on trunk, r=jgraham
https://hg.mozilla.org/integration/autoland/rev/c5e8ce62436d
[ci] web-platform-test-reftest: Run fission linux64-asan/opt on trunk, r=jgraham
https://hg.mozilla.org/integration/autoland/rev/8adbfb955b40
[ci] web-platform-test-print-reftest: Run fission windows10-qr/debug on trunk, r=jgraham
https://hg.mozilla.org/integration/autoland/rev/7bd2b5fc9b37
[ci] web-platform-test-print-reftest: Run fission linux64-asan/opt on trunk, r=jgraham

Comment 16

[Bogdan Tara[:bogdan_tara | bogdant]]

https://hg.mozilla.org/mozilla-central/rev/80e6aef40e50
https://hg.mozilla.org/mozilla-central/rev/5991f2ed9496
https://hg.mozilla.org/mozilla-central/rev/5802c0beafd1
https://hg.mozilla.org/mozilla-central/rev/a62f976a393a
https://hg.mozilla.org/mozilla-central/rev/d9de90fb0b90
https://hg.mozilla.org/mozilla-central/rev/c5e8ce62436d
https://hg.mozilla.org/mozilla-central/rev/8adbfb955b40
https://hg.mozilla.org/mozilla-central/rev/7bd2b5fc9b37

Comment 17

[Andrew Halberstadt [:ahal]]

Attachment: Bug 1696042 - [ci] web-platform-test-wdspec: Run fission windows10-qr/debug on trunk, r?jgraham

This enables the following configuration on autoland/central:

test-windows10-64-qr/debug-web-platform-tests-wdspec-fis-e10s

Comment 18

[Andrew Halberstadt [:ahal]]

Attachment: Bug 1696042 - [ci] web-platform-test-wdspec: Run fission linux64-asan/opt on trunk, r?jgraham

This enables the following configuration on autoland/central:

test-linux1804-64-asan/opt-web-platform-tests-wdspec-fis-e10s

Depends on D111141

Comment 19

[Andrew Halberstadt [:ahal]]

Attachment: Bug 1696042 - [ci] web-platform-test-wdspec-headless: Run fission linux64-asan/opt on trunk, r?jgraham

This enables the following configurations on autoland/central:

test-linux1804-64-asan/opt-web-platform-tests-wdspec-headless-e10s
test-linux1804-64-asan/opt-web-platform-tests-wdspec-headless-fis-e10s

Depends on D111142

Comment 20

[Andrew Halberstadt [:ahal]]

Attachment: Bug 1696042 - [ci] web-platform-test-wdspec-headless: Run fission windows10-qr/debug on trunk, r?jgraham

Depends on D111143

Comment 21

[Andrew Halberstadt [:ahal]]

Still some more work to do here, almost done.

Comment 22

[Andrew Halberstadt [:ahal]]

I'll do another try run in a bit, but tsan looks a bit flaky here as well. Might be common issues with the other suites, in which case it would be better to tackle all tsan tasks in a separate bug.

Comment 23

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/6fa186aef599
[ci] web-platform-test-wdspec: Run fission windows10-qr/debug on trunk, r=jgraham
https://hg.mozilla.org/integration/autoland/rev/089853a337cd
[ci] web-platform-test-wdspec: Run fission linux64-asan/opt on trunk, r=jgraham
https://hg.mozilla.org/integration/autoland/rev/4a56ecca58bd
[ci] web-platform-test-wdspec-headless: Run fission linux64-asan/opt on trunk, r=jgraham

Comment 24

[Narcis Beleuzu [:NarcisB]]

https://hg.mozilla.org/mozilla-central/rev/6fa186aef599
https://hg.mozilla.org/mozilla-central/rev/089853a337cd
https://hg.mozilla.org/mozilla-central/rev/4a56ecca58bd

Comment 25

[Andrew Halberstadt [:ahal]]

There was one patch left to land, but I think I'll call this done after that.

Chris, fyi there are still some issues with tsan:
https://treeherder.mozilla.org/jobs?repo=try&revision=e2487391f036693313bd60adfb1ce49516ba959f&searchStr=tsan

Probably some root causes in common with the mochitest problems. We could probably green it up through manifest annotations, but I think I'll delay doing that in favour of working on the requests in the other bugs for now.

Maybe there could be a single fission tsan bug to enable it across all the desired suites (since I suspect each suite is hitting similar problems).

Comment 26

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/35c54fcf3396
[ci] web-platform-test-wdspec-headless: Run fission windows10-qr/debug on trunk, r=jgraham,webdriver-reviewers

Comment 27

[Natalia Csoregi [:nataliaCs]]

https://hg.mozilla.org/mozilla-central/rev/35c54fcf3396

Comment 28

[Chris Peterson [:cpeterson]]

(In reply to Andrew Halberstadt [:ahal] from comment #25)

Chris, fyi there are still some issues with tsan:
https://treeherder.mozilla.org/jobs?repo=try&revision=e2487391f036693313bd60adfb1ce49516ba959f&searchStr=tsan

Probably some root causes in common with the mochitest problems. We could probably green it up through manifest annotations, but I think I'll delay doing that in favour of working on the requests in the other bugs for now.

Maybe there could be a single fission tsan bug to enable it across all the desired suites (since I suspect each suite is hitting similar problems).

I peeked at the tsan failures. Like the tsan failures in mochitest-plain bug 1698193, the failures are nearly all timeouts. The tsan tests are slow and Fission will make them slower. Like bug 1698193, the Fission team is not concerned about these timeouts, since they don't indicate tsan bugs. I will file a new bug to skip those tsan tests and enable the rest for Fission: bug 1706560.

Comment 29

[Phabricator Automation]

Comment on attachment 9213104 [details]
WIP: Bug 1696042 - [ci] web-platform-test: Run fission linux64-tsan on trunk

Revision D110555 was moved to bug 1706560. Setting attachment 9213104 [details] to obsolete.