1244176 - On try, allow certain test jobs to *not* be scheduled unless -u <job_name> is specified

Status: RESOLVED FIXED

(Firefox Build System :: Task Configuration, task)

Description

[Armen [:armenzg]]

-u all will schedule all test jobs.
Having a way to indicate that certain jobs are *only* scheduled when specified explicitely is a feature wanted.

For various reasons:
* There is a limited pool for that platform (e.g. Mac)
 * For instance, on Buildbot, we don't run on 10.6 unless specified (notice that this is a platform example rather than a specific job)
* The job is not ready for prime time
 * even though the developer could carry a patch around
* We need to run that job once in a while to simply grab some artifacts from it

This is a similar request to bug 1116275.

Comment 1

[Greg Arndt [:garndt]]

I would need to double check, but I think that as long as the flag is not listed in testing/taskcluster/tasks/branches/base_job_flags.yml it won't be scheduled with "all".  Perhaps I'm mistaken on that one so I would need to validate that.

Comment 2

[Armen [:armenzg]]

The request is to not need to touch base_job_flags.yml (or the try version of it) but simply specify the job in the try syntax.

For the record, Buildbot does this for build jobs but not for test jobs.

Comment 3

[Armen [:armenzg]]

This is specially important for tests like valgrind which triggers 40 chunks which can last easily 5-8 hours.
We only want to schedule it when specifically requested in the try syntax.

Comment 5

[Dustin J. Mitchell [:dustin] (he/him)]

I'm pretty uncomfortable with the idea of "all" not meaning "all".  Maybe this is something that can be included in Andrew's thinking about a new interface to try?

Comment 6

[Andrew Halberstadt [:ahal]]

Sure, we could block on try parser moving client side if we want. But I think it'll be roughly the same amount of work either way.

I do agree with Armen that this is something we need though. I think "all" should mean "all jobs that I need to run to make sure I don't get backed out". But there can be jobs other jobs that don't fit that criterion. For example, jobs that are still being worked on (and only scheduled on try), jobs that don't test anything but which people might wish to run from time to time (e.g code coverage).

Comment 7

[Dustin J. Mitchell [:dustin] (he/him)]

I'd like to hold off on this until we have retired the legacy task, at which point we can introduce a 'try_by_default' attribute or something like that.

Comment 8

[Geoff Brown [:gbrown]]

(In reply to Armen Zambrano [:armenzg] - Engineering productivity from comment #0)
> For various reasons:

In case we need more incentive, I'll mention:

* A test, or build, is being retired and the phase-out is riding the trains. As long as the job runs on aurora/beta/release, we need it available on try; but if the job is accidentally included via try -all and fails (since it isn't normally run on trunk any longer), developer confusion results. See bug 1184117. (Incidentally, I like Ryan's terminology there: "opt-in by default on try".)

* PGO may be a special case. Historically, pgo on try required a special patch - https://wiki.mozilla.org/ReleaseEngineering/TryChooser#What_if_I_want_PGO_for_my_build - but that's no longer required with taskcluster. PGO builds are expensive (long running) and it's not clear that they provide much value to the typical try -all push. See also bug 1274022.

I think it would be great if the task yml included a try_by_default (try_opt_in_only?) attribute.

Comment 9

[Dustin J. Mitchell [:dustin] (he/him)]

I'd be fine seeing that attribute added and hacking support for it into the legacy kind and try_option_parser.

Comment 10

[Selena Deckelmann :selenamarie :selena]

Very much in favor of "try_by_default" rather than "all". Thank you.

Comment 11

[Dustin J. Mitchell [:dustin] (he/him)]

My thinking is to add a "try-by-default" tag to a test suite and reflect it into the attributes.  Then build a target task method that filters for that attribute.  This could be done for tests once bug 1281004 lands.  Not that I'm promising to do it!

Worth noting, we will need flexible ways to control whether a particular job runs on *any* branch by default, so perhaps the syntax should be something like

  run-for-projects:
    - try
    - mozilla-central

but we need to think about both including ("run only on inbound and try") or excluding ("run everywhere but inbound").

Comment 12

[Gregory Szorc [:gps]]

My gut reaction is "run for projects" is a bad idea because projects/repos are just names given to DAG heads. I would prefer we run things based on what changes. I want us to get to a world where we can e.g. do a build with a beta configuration on a revision currently on try or central.

Comment 13

[Dustin J. Mitchell [:dustin] (he/him)]

How would "a build with a beta configuration" be triggered based on what changes?

Comment 14

[Gregory Szorc [:gps]]

(In reply to Dustin J. Mitchell [:dustin] from comment #13)
> How would "a build with a beta configuration" be triggered based on what
> changes?

It would probably be a manual thing via the Treeherder web UI or special Try syntax.

This type of thing is typically only needed by a few people. e.g. build system maintainers would like the ability to test beta or release builds or l10n jobs because we often change things in central and don't find out until 2 months later that it broke some process we only run on the beta or release channels.

Comment 15

[Dustin J. Mitchell [:dustin] (he/him)]

We have `run-on-projects` for non-test tasks; see bug 1301762.  This doesn't work for tests right now, only because none of the target-tasks methods expect it there.  That could easily be changed to support this behavior for test jobs.

Comment 16

[Dustin J. Mitchell [:dustin] (he/him)]

run-on-projects works for tests now.