1543533 - push with try syntax "-u all" in commit message should only schedule jobs which have 'try' or 'all' in run-on-projects

Status: NEW

(Firefox Build System :: Task Configuration, defect)

Description

[Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)]

At the moment, a push with a try syntax containing "-u all" will schedule all tests in the full task graph. Taking the "run-on-projects" attribute into account where it contains 'try' or 'all'

From https://searchfox.org/mozilla-central/rev/dd7e27f4a805e4115d0dbee70e1220b23b23c567/taskcluster/docs/attributes.rst#22

run_on_projects
===============

The projects where this task should be in the target task set.  This is how
requirements like "only run this on inbound" get implemented.  These are
either project names or the aliases

 * `integration` -- integration repositories (autoland, inbound, etc)
 * `trunk` -- integration repositories plus mozilla-central
 * `release` -- release repositories including mozilla-central
 * `all` -- everywhere (the default)

For try, this attribute applies only if ``-p all`` is specified.  All jobs can
be specified by name regardless of ``run_on_projects``.

If ``run_on_projects`` is set to an empty list, then the task will not run
anywhere, unless its build platform is specified explicitly in try syntax.

That will remove the following jobs:
test-android-em-7.0-x86_64/debug-marionette
test-android-em-7.0-x86_64/opt-marionette
test-linux32/debug-mochitest-webgl1-core
test-linux64-devedition/opt-mochitest-valgrind-$chunk
test-linux64-shippable/opt-mochitest-valgrind-$chunk
test-linux64/debug-crashtest-sw-e10s
test-linux64/debug-jsreftest-sw-e10s-$chunk
test-linux64/debug-mochitest-a11y-sw
test-linux64/debug-mochitest-browser-chrome-sw-e10s-$chunk
test-linux64/debug-mochitest-chrome-sw-$chunk
test-linux64/debug-mochitest-clipboard-sw-e10s
test-linux64/debug-mochitest-devtools-chrome-sw-e10s-$chunk
test-linux64/debug-mochitest-gpu-sw-e10s
test-linux64/debug-mochitest-media-sw-e10s-$chunk
test-linux64/debug-mochitest-plain-headless-sw-e10s-$chunk
test-linux64/debug-mochitest-sw-e10s-$chunk
test-linux64/debug-mochitest-webgl1-core-sw-e10s
test-linux64/debug-mochitest-webgl1-ext-sw-e10s
test-linux64/debug-reftest-no-accel-sw-e10s-$chunk
test-linux64/debug-reftest-sw-e10s-$chunk
test-linux64/debug-test-verify-wpt-sw-e10s
test-linux64/debug-web-platform-tests-reftests-sw-e10s-$chunk
test-linux64/debug-web-platform-tests-sw-e10s-$chunk
test-linux64/debug-web-platform-tests-wdspec-sw-e10s-$chunk
test-linux64/debug-xpcshell-sw-$chunk
test-linux64/opt-mochitest-valgrind-$chunk
test-windows10-64-shippable/opt-browser-instrumentation-e10s

That lowers the amount of jobs for "-u all" from 2598 to 2340 tasks.

Comment 1

[Geoff Brown [:gbrown]]

test-android-em-7.0-x86_64/debug-marionette
test-android-em-7.0-x86_64/opt-marionette

Why would these 2 be affected? I don't see anything special about their run-on-projects.

Comment 2

[Dustin J. Mitchell [:dustin] (he/him)]

"should" is a funny word about try syntax - everyone believes different "should"s and so bugs like this tend to regress bugs from six months ago saying the opposite, and be regressed by another bug saying the opposite later. If -u all doesn't mean "all", then if you really want to run all tests do you write -u all_no_i_really_mean_it_this_time?

The escape hatch from this sort of cycle is tryselect and the various ./mach try options. I believe that already supports many of the common cases (such as "please run all of the stuff that's scheduled to run on try by default") and can be customized using presets for the less-common cases. If there's a common case it doesn't support well, then we should add functionality there.

Comment 3

[Tom Prince [:tomprince]]

While I agree that the solution to this is better tools like mach try chooser and mach try fuzzy, until we get rid of try syntax, we should reduce the amount of things it runs, particularly when it used indiscriminately like -u all. If there are configurations that are explicitly annotated to not run on try, then we shouldn't run them if people don't explicitly ask for them.

Comment 4

[Mike Hommey [:glandium]]

I also think a reading of "all" meaning "everything that might blow up in my face when landing" is the most reasonable meaning of "all". Who really cares that "all" runs obscure things that don't even run on central or integration branches (which are tier 1 but shouldn't even be in the first place)? Few people.