1888967 - Consider Preset Try Options for GeckoView

Status: RESOLVED FIXED

(Developer Infrastructure :: Try, enhancement)

Description

[Olivia Hall [:olivia]]

Now with the monorepo, other areas of android have ./mach try --presets of fenix firefox-android, android-components and focus. It would be nice to have a similar set of default preset tests for geckoview that cover the basic junit and other core tests.

Comment 1

[Olivia Hall [:olivia]]

(This bug should likely move to a tooling component, but need a list of tests for the preset first.)

Comment 2

[Olivia Hall [:olivia]]

Hi, I'm not sure of the right component this should go to, but GeckoView developers would like a preset of geckoview to use with ./mach try --preset geckoview that runs the following tests to help standardize test selection:

[
"source-test-mozlint-android-lints",
"source-test-mozlint-eslint",
"source-test-mozlint-rejected-words",
"source-test-mozlint-file-perm",
"source-test-mozlint-file-whitespace",
"test-android-em-7.0-x86_64-qr/opt-geckoview-test-verify",
"test-android-em-7.0-x86_64-qr/debug-isolated-process-geckoview-junit-nofis",
"test-android-em-7.0-x86_64-qr/debug-isolated-process-geckoview-mochitest-plain-5",
"test-android-em-7.0-x86_64-qr/opt-geckoview-mochitest-plain-5",
"test-android-em-7.0-x86_64-qr/debug-isolated-process-geckoview-mochitest-media-2",
"test-android-em-7.0-x86_64-qr/debug-isolated-process-geckoview-mochitest-media-1",
"test-android-em-7.0-x86_64-qr/debug-isolated-process-geckoview-test-verify-nofis",
"test-android-em-7.0-x86_64-qr/debug-isolated-process-geckoview-xpcshell-2",
"test-android-em-7.0-x86_64-qr/opt-geckoview-test-verify-nofis",
"test-android-em-7.0-x86_64-qr/debug-isolated-process-geckoview-cppunittest-1proc",
"test-android-em-7.0-x86_64-qr/opt-geckoview-mochitest-plain-3",
"test-android-em-7.0-x86_64-qr/opt-geckoview-mochitest-plain-4",
"test-android-em-7.0-x86_64-qr/debug-geckoview-xpcshell-1",
"test-android-em-7.0-x86_64-qr/opt-geckoview-cppunittest-1proc",
"test-android-em-7.0-x86_64-qr/debug-isolated-process-geckoview-mochitest-plain-1",
"test-android-em-7.0-x86_64-qr/debug-isolated-process-geckoview-mochitest-plain-gpu",
"test-android-em-7.0-x86_64-qr/opt-geckoview-junit-nofis-ship",
"test-android-em-7.0-x86_64-qr/debug-isolated-process-geckoview-mochitest-plain-4",
"test-android-em-7.0-x86_64-qr/debug-isolated-process-geckoview-test-verify",
"test-android-em-7.0-x86_64-qr/debug-isolated-process-geckoview-mochitest-plain-2",
"test-android-em-7.0-x86_64-qr/debug-isolated-process-geckoview-junit-fis",
"test-android-em-7.0-x86_64-qr/opt-geckoview-junit-nofis",
"test-android-em-7.0-x86_64-qr/opt-geckoview-gtest-1proc",
"test-android-em-7.0-x86_64-qr/debug-isolated-process-geckoview-xpcshell-1",
"test-android-em-7.0-x86_64-qr/debug-isolated-process-geckoview-junit-nofis-ship",
"test-android-em-7.0-x86_64-qr/opt-geckoview-junit-fis",
"test-android-em-7.0-x86_64-qr/opt-geckoview-junit",
"test-android-em-7.0-x86_64-qr/debug-geckoview-xpcshell-2",
"test-android-em-7.0-x86_64-qr/opt-geckoview-mochitest-plain-1",
"test-android-em-7.0-x86_64-qr/debug-isolated-process-geckoview-junit",
"test-android-em-7.0-x86_64-qr/debug-isolated-process-geckoview-mochitest-plain-3",
"test-android-em-7.0-x86_64-qr/opt-geckoview-mochitest-plain-2",
"test-android-em-7.0-x86_64-qr/opt-geckoview-mochitest-plain-gpu",
"test-android-em-7.0-x86_64-qr/debug-isolated-process-geckoview-gtest-1proc",
"test-android-em-7.0-x86_64-qr/opt-geckoview-mochitest-media"
]

Comment 3

[Johan Lorenzo [:jlorenzo]]

Good news! It should be a matter of updating tools/tryselect/try_presets.yml[1] just like we did in bug 1882616. I'm happy to help reviewing if the change is urgent 🙂

[1] https://searchfox.org/mozilla-central/rev/294e1fbdcc9ca0c328c372392e03bb49df4ee77e/tools/tryselect/try_presets.yml#94

Comment 4

[Heitor Neiva [:hneiva]]

Attachment: Bug 1888967 - Add try presets for android-geckoview r?#releng,ohall,jlorenzo

Comment 5

[Heitor Neiva [:hneiva]]

:olivia can you please double check if these are the tasks required to run under the new profile?

https://treeherder.mozilla.org/jobs?repo=try&revision=024097a638adf039ffa0cbd2b298aa157cab3771&selectedTaskRun=cvFGuu7wSvW-ttgMe11XUg.0

Comment 6

[Olivia Hall [:olivia]]

Awesome, thank you!

• Could you move X tests (xpcshell) to Android 7.0 x86-64 WebRender opt and Android 7.0 x86-64 WebRender debug-isolated-process? (It is okay there, but might be nice to avoid a build, if we can.)
• Could we skip Linux 18.04 x64 opt and Windows 11 x64 opt items too?
• It would probably be nice to avoid some of the builds on Android 5.0 x86-64 opt, Android 5.0 x86-64 debug, Android 5.0 x86-64 debug-isolated-process, and Android 7.0 x86-64 WebRender debug for a time saving opportunity. I'm not sure what is typical though for a general test though and a best practice, what do you recommend?

I'll confirm this with other GeckoView developers too, thanks!

:owlish, :calu, anything you'd like to add or change or recommend?

Comment 7

[Geoff Brown [:gbrown]]

Just checking: No reftests or web-platform-tests are desired with this preset?

Comment 8

[Olivia Hall [:olivia]]

Just checking: No reftests or web-platform-tests are desired with this preset?

There was some concern with GeckoView reviewers and time to run WPT tests. If you think it would be a best practice and not a huge time/cost concern, then it could be nice to add.

I personally usually run WPT tests for certain changes too. Not sure what the right balance of test completeness and time/cost we should be aiming for.

Comment 9

[Geoff Brown [:gbrown]]

There's probably no "right" answer. I just wanted to check those tests weren't being accidentally overlooked.

Comment 10

[Heitor Neiva [:hneiva]]

:olivia

Could you move X tests (xpcshell) to Android 7.0 x86-64 WebRender opt and Android 7.0 x86-64 WebRender debug-isolated-process? (It is okay there, but might be nice to avoid a build, if we can.)

I'll update the xpcshell tasks to run from opt. I'll post a new run soon here.

Could we skip Linux 18.04 x64 opt and Windows 11 x64 opt items too?

some of these tasks only ran because of the nature of changes that I had on the push. For example the windows and linux jobs you mentioned ran because of changes in the tryselect itself [1]
That being said, there are some lint/test tasks that "always" run as they are "cheap" enough to run on pretty much every push.

It would probably be nice to avoid some of the builds on Android 5.0 x86-64 opt, Android 5.0 x86-64 debug, Android 5.0 x86-64 debug-isolated-process, and Android 7.0 x86-64 WebRender debug for a time saving opportunity. I'm not sure what is typical though for a general test though and a best practice, what do you recommend?

The Android 5.0 builds seem to be required for some tooling tasks [2]. If we still want those to be avoided, I'd suggest we follow up in a separate issue to make the tooling tasks not depend on those.

[1] https://searchfox.org/mozilla-central/source/taskcluster/ci/source-test/python.yml#365
[2] https://firefox-ci-tc.services.mozilla.com/tasks/J64ITo6eT7G_4wah6y-_9Q

Comment 11

[Olivia Hall [:olivia]]

Thanks you so much!

some of these tasks only ran because of the nature of changes that I had on the push.

Ohh, that makes sense, thanks for explaining!

The Android 5.0 builds seem to be required for some tooling tasks [2]. If we still want those to be avoided, I'd suggest we follow up in a separate issue to make the tooling tasks not depend on those.

Sounds good, I don't have a strong preference! Moving away from Android 5.0 long term would probably make sense though.

Comment 12

[Heitor Neiva [:hneiva]]

New run: https://treeherder.mozilla.org/jobs?repo=try&revision=1239ab1bd433014ab114ea74296e34e5078755e7

Comment 13

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/3626f12014e3
Add try presets for android-geckoview r=ohall,releng-reviewers,gbrown

Comment 14

[tszentpeteri]

https://hg.mozilla.org/mozilla-central/rev/3626f12014e3

Comment 15

[[:owlish] 🦉 PST]

Hey folks, can we please add these builds in the preset:

• Android 7.0 x86-64 WebRender debug
• Android 7.0 x86-64 Lite WebRender opt
• Android 7.0 x86-64 Lite WebRender Shippable
• Android 7.0 x86-64 Shippable WebRender

Comment 16

[Julien Cristau [:jcristau]]

Please file a new bug for this.