1670995 - make android webrender the default for unittests

Status: RESOLVED FIXED

(Testing :: General, task)

Description

[Joel Maher ( :jmaher ) (UTC -8)]

in bug 1670284 there is a request to do this for performance tests, this bug is used for unittests.

here is a list of jobs/tests that are only run on !qr:
emulator debug/opt:

• cppunit
• xpcshell
• web-platform-tests

pixel2 api-16 builds debug/shippable:

• jittest
• jsreftest

pixel2 aarch64 builds debug/shippable:

• mochitest-media
• jittest
• mochitest-webgl
• mochitest-media-spi

questions:

• is the intention to not run non webrender tests?
• is the intention to run everything in webrender?
• what about tests that don't make sense? cppunit, xpcshell, jittest, jsreftest?

:ktaeleman can you help answer the questions

Comment 1

[Kris Taeleman (:ktaeleman)]

Thanks for filing :jmaher!
I think :jnicol has a better overview to answer these questions.

Comment 2

[Jamie Nicol [:jnicol]]

Hi Joel,

is the intention to not run non webrender tests?

No, we plan to run both while we ship both webrender and non-webrender. But with increasing webrender usage (planning to be the majority of users on 84) we need better webrender coverage (by reducing non-webrender coverage if needs be).

is the intention to run everything in webrender?

Everything that makes sense to. I'm afraid I don't have a great understanding about what most of the suites are.

what about tests that don't make sense? cppunit, xpcshell, jittest, jsreftest?

If these don't make any sense to run with webrender, then that's fine.

emulator debug/opt:

• web-platform-tests

These probably would make sense to run. (the reftests certainly).

pixel2 aarch64 builds debug/shippable:

• mochitest-media
• mochitest-webgl
• mochitest-media-spi

These would be great to run. I imagine there would probably be some failures as things stand, which we will need to fix.

Currently the tests that do run on Pixel 2s are tier 2, and for reftests I believe we run a reduced suite. It would be good if they could be tier 1.

Comment 3

[Joel Maher ( :jmaher ) (UTC -8)]

I see TODO items as:

1. emulator web-platform-tests as webrender
2. phone media/webgl tests as webrender
3. making android pixel reftests tier-1

#3 should be doable- keep in mind that we have limited hardware and that is the reason for having them as tier-2. We will be reducing hardware as we diversify the hardware pool.

as for #2, I would vote for replacement, not duplicating.

as for #1, I think we should try for duplication.

2 questions:

1. Any concerns with the recommendations for 1 and 2?
2. if we find a lot of failed tests, can we just disable and get things up asap? we can spend the 2 weeks filing bugs after things are online

Comment 4

[Joel Maher ( :jmaher ) (UTC -8)]

a quick update on how things look for #1 and #2:
https://treeherder.mozilla.org/#/jobs?repo=try&revision=7e699bb2c57700994b0113a5e7d1856fa6c6bdc1

mochitest media runs great in webrender, that should be straightforward to switch over. The web-platform-tests have a handful of failures, I will wait for all results and look at the manifests, possibly we could have known failures on webrender and I would just automatically move those over to android. Also that try push ran wpt-fis tests which are not enabled by default, so more orange than would show up by default.

Comment 5

[Jamie Nicol [:jnicol]]

Sounds good.

Any concerns with the recommendations for 1 and 2?

Only a slight concern about not doing any testing for media or webgl for non-webrender. Would it be possible to still run them as tier 2 / less frequently? I understand resources are scarce, and would definitely rather have them running for webrender than non-webrender if we cannot do both.

if we find a lot of failed tests, can we just disable and get things up asap? we can spend the 2 weeks filing bugs after things are online

Sounds like a good approach. And yes, the tests look in a better state currently than I was expecting. Pleasantly surprised :)

Comment 6

[Joel Maher ( :jmaher ) (UTC -8)]

Attachment: Bug 1670995 - adjust android unittests to be -qr. r=bc

adjust android unittests to be -qr

Comment 7

[Joel Maher ( :jmaher ) (UTC -8)]

I have one patch up for the hardware and regular web-platform-tests; I should have a patch today for the web-platform-test-reftests (a lot to adjust).

We might need to refine this for getting scheduling proper on the hardware, that can be done after we land.

:jnicol: This isn't making the cutoff for the beta merge day (today) to be in Firefox 83, is that desired?

Comment 8

[Jamie Nicol [:jnicol]]

Thanks Joel that's great.

84 is perfect, that's where we'll be riding the trains for having webrender enabled for the majority of users.

Comment 9

[Joel Maher ( :jmaher ) (UTC -8)]

Attachment: Bug 1670995 - android -qr wpt-reftests. r=jnicol

run android web-platform-test-reftests on -qr

Comment 10

[Joel Maher ( :jmaher ) (UTC -8)]

Attachment: Bug 1670995 - run android mochitest/wpt on -qr by default. r=jnicol,bc

run android mochitest/wpt on -qr by default

Comment 11

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/54f221c0ff5c
run android mochitest/wpt on -qr by default. r=jnicol

Comment 12

[Sandor Molnar[:smolnar]]

https://hg.mozilla.org/mozilla-central/rev/54f221c0ff5c