1700454 - Add a shared try preset for geckodriver

Status: RESOLVED FIXED

(Testing :: geckodriver, task, P3)

Description

[Henrik Skupin [:whimboo][⌚️UTC+1]]

Similar to bug 1667325 where we added a try preset for Marionette we also need one for geckodriver.

Jobs that would be needed:

• Rust unit tests
• Wdspec tests
• Subset of Browsertime tests (desktop, and Android)

It needs to be added to https://searchfox.org/mozilla-central/source/tools/tryselect/try_presets.yml.

Greg, are there some particularly short-running browsertime jobs that we could use for smoketesting that changes to geckodriver are fine?

Comment 1

[Greg Mierzwinski [:sparky]]

The amazon test on the P2, and on Linux should cover most of our use cases. If you want to check for performance changes, this set of high-value tests could be used:

Desktop: https://searchfox.org/mozilla-central/source/taskcluster/ci/test/browsertime-desktop.yml#82-89
Mobile: https://searchfox.org/mozilla-central/source/taskcluster/ci/test/browsertime-mobile.yml#169-175

Comment 2

[Henrik Skupin [:whimboo][⌚️UTC+1]]

There are a lot of Amazon test flavors. So would the following perhaps match what we should run on Android?

'browsertime 'p2 'fenix !vismet 'aarch64 !live !wr 'amazon !search

Performance might be good too, so we might only have to change the above to 'vismet, and instead use 'amazon-search? What's the difference in test execution time? Probably we want to run the one that is shorter.

Comment 3

[Greg Mierzwinski [:sparky]]

Ah yes there's the amazon and amazon-search tests, the plain amazon is enough. The query looks good, I would add 'ship to it so you only run the shippable tests:

'browsertime 'p2 'fenix 'aarch64 !live !wr 'amazon !search 'ship

You may as well run vismet on them since we always take recordings in the test task and the vismet task only takes about 2-5mins to run.

Comment 4

[Henrik Skupin [:whimboo][⌚️UTC+1]]

Attachment: Bug 1700454 - [geckodriver] Add geckodriver try preset.

Comment 5

[Henrik Skupin [:whimboo][⌚️UTC+1]]

I just pushed a try build:
https://treeherder.mozilla.org/jobs?repo=try&revision=8ba6936e86cd768bbaffa60e5ebf6871844ae999

But as it looks like it doesn't trigger the Browsertime jobs on Android that need --full to be specified.

Andrew, is there a way to get these integrated too?

Comment 6

[Henrik Skupin [:whimboo][⌚️UTC+1]]

As it looks like Browsertime tests are missing completely even on desktop where full it not required.

Comment 7

[Andrew Halberstadt [:ahal]]

Are these the ones that are disabled due to resource constraints? If so try adding disable_target_task_filter: true to your preset. The values just get forwarded in to the run function.

Comment 8

[Henrik Skupin [:whimboo][⌚️UTC+1]]

Thank you Andrew. I tested that but it actually didn't make a difference. But, I saw the full option for the run function, and noticed that this one is actually required locally when triggering Browsertime jobs. Changing it accordingly made it work. I will push that change and request review.

Comment 9

[Henrik Skupin [:whimboo][⌚️UTC+1]]

Greg, I'm having a problem with the preset and the browsertime selection. By default those tests aren't run on Android and as such are only available when specifying --full. But that option is not wanted in the try preset.

What specifically blocks us from running (even only some particular) browsertime jobs on autoland/central? As you suggested I added the Amazon search vismet ones.

Thanks.

Comment 10

[Greg Mierzwinski [:sparky]]

:whimboo, the --full flag allows you to see tasks with empty run-on-projects and any other tasks that would have been filtered out. There are some tests that should appear when --full is not provided for android, however, in this case, the android browsertime tasks are prevented from running on try by this filter: https://searchfox.org/mozilla-central/source/taskcluster/taskgraph/target_tasks.py#30

The reason we've prevented it from running when --full isn't provided is because many developers run everything they can which caused large backlogs on android. I don't want to remove this restriction though since we'll likely hit the same issue again.

Comment 11

[Henrik Skupin [:whimboo][⌚️UTC+1]]

Ok, that sounds fair. But would mean that we aren't able to test any BrowserTime job on Android. How long would it actually take until a regression is noticeable?

I think before a geckodriver release we should at least run this specific BrowserTime job to ensure that this test job works.

If that is ok with you I will remove the BrowserTime entry.

Comment 12

[Greg Mierzwinski [:sparky]]

We'd see a performance regression within about 1-3 days and we'd see issues in running the tasks within 1 day (the longest being the fenix tests which run once a day).

Keeping out browsertime is fine for me, but you could leave the desktop test in it which would help catch some functional issues.

Comment 13

[Henrik Skupin [:whimboo][⌚️UTC+1]]

(In reply to Greg Mierzwinski [:sparky] from comment #12)

Keeping out browsertime is fine for me, but you could leave the desktop test in it which would help catch some functional issues.

Ok, thank you for the feedback. Lets get it done this way then.

Comment 14

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/9e267c6cbc39
[geckodriver] Add geckodriver try preset. r=webdriver-reviewers,ahal,sparky,jdescottes DONTBUILD

Comment 15

[Natalia Csoregi [:nataliaCs]]

https://hg.mozilla.org/mozilla-central/rev/9e267c6cbc39