1856473 - Add try presets for WebExtensions tests

Status: RESOLVED FIXED

(Developer Infrastructure :: Try, task)

Description

[Rob Wu [:robwu]]

./mach try auto has been very unreliable for the WebExtensions team (e.g. bug 1830760 and many others).

We have either been running too many tests ("just in case"), which is expensive, or too few (followed by backouts).

I have experimented with some queries last weekend and found a sweet spot between running too many tests and too few, so I'll attach the preset here.

Comment 1

[Rob Wu [:robwu]]

Attachment: Bug 1856473 - Add webextensions-tests preset

The path of GeckoView extension tests 1 are not included in the query,
because they would not be run anyway (bug 1826909).

Comment 2

[Rob Wu [:robwu]]

I have linked the new preset (--preset webextensions) from https://wiki.mozilla.org/WebExtensions/Try_Server

Note: the in-tree preset file did not include paths before, and my patch is the first to use paths (with a work-around for bug 1556445; if that bug gets fixed we may simplify the preset).

Comment 3

[Rob Wu [:robwu]]

Not sure why addons-jira whiteboard isn't working; adding link manually.

Comment 4

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/2f99ebdee682
Add webextensions-tests preset r=extension-reviewers,zombie

Comment 5

[Natalia Csoregi [:nataliaCs]]

https://hg.mozilla.org/mozilla-central/rev/2f99ebdee682

Comment 6

[Rob Wu [:robwu]]

Here is a comparison between the scheduled tasks with the preset, and before I introduced the preset.
The new preset runs relevant extension tests only, on all platforms (Windows, Linux, macOS, Android).
The old preset runs many unrelated tests, on Linux only (and misses some relevant tests).
Thanks to the new preset, the CI machine time per try push used dropped from 6 days down to 1½ day (75% CI time saved).

$  ./mach try --preset webextensions --no-push
Artifact builds enabled, pass --no-artifact to disable
Task configuration changed, generating target task set
estimates: Runs 134 tasks (95 selected, 39 dependencies)
estimates: Total task duration 1 day, 13:12:10
estimates: In the top 36% of durations
estimates: Should take about 1:28:48 (Finished around 2023-12-08 16:03)
Commit message:
Fuzzy query='64-qr/ | 'windows11-64-2009-qr !wpt !gpu !msix&paths=browser/components/extensions/test/:mobile/android/components/extensions/test/:toolkit/components/extensions/test/:toolkit/mozapps/extensions/test/

Compared with the previous one that was in use in the team:

$ ./mach try fuzzy --query="'linux '64-qr/ "'!'"asan 'mochitest "'!'"-webgl "'!'"-webgpu- "'!'"-remote- "'!'"-media-" --query="'linux '64-qr/ "'!'"asan 'xpcshell" --query="'linux '64-qr/ "'!'"asan 'test-verify "'!'"gpu "'!'"wpt" --no-push
Artifact builds enabled, pass --no-artifact to disable
estimates: Runs 425 tasks (411 selected, 14 dependencies)
estimates: Total task duration 6 days, 1:03:36
estimates: In the top 12% of durations
estimates: Should take about 1:25:48 (Finished around 2023-12-08 16:15)
Commit message:
Fuzzy query='linux '64-qr/ !asan 'mochitest !-webgl !-webgpu- !-remote- !-media-&query='linux '64-qr/ !asan 'xpcshell&query='linux '64-qr/ !asan 'test-verify !gpu !wpt

The command above is the self-contained equivalent to the following preset that was in use in the team before:

# Usage: ./mach try --preset linux64-mochitests-xpcshells-testverify
linux64-mochitests-xpcshells-testverify:
  query:
  - '''linux ''64-qr/ !asan ''mochitest !-webgl !-webgpu- !-remote- !-media-'
  - '''linux ''64-qr/ !asan ''xpcshell'
  - '''linux ''64-qr/ !asan ''test-verify !gpu !wpt'
  selector: fuzzy