1397722 - [tryselect] Consider excluding code coverage runs from default set of tasks

Status: RESOLVED FIXED

(Developer Infrastructure :: Try, enhancement, P3)

Description

[Andrew Halberstadt [:ahal]]

Currently pgo and code coverage builds and tests are part of the target set, so show up when running |mach try fuzzy|. I think we should consider pre-filtering these tasks out so people have to run |mach try fuzzy --full| to be able to select them.

I think most people don't particularly care about running on these platforms, but they often get run anyway when people use <ctrl-a>. Personally almost all of my queries start with "!pgo !cov" to filter them out.

For anyone who *does* want to run pgo or code coverage, this just means they'd need to pass in --full. If desired, they can even alias this as a default.

Maybe instead of filtering these out by name, we could filter out tasks that run periodically on central or something.

Comment 1

[Greg Mierzwinski [:sparky]]

When we were using try option syntax we kept jsdcov/ccov out of '-p all -u all' pushes through the 'run-on-projects'. Is this the same sort problem? I ask because I'm not sure what |mach try fuzzy| is equivalent to in try option syntax.

You're right though that most people never need to have ccov/jsdcov scheduled by default (and it's a waste of resources).

Comment 2

[Andrew Halberstadt [:ahal]]

Yeah this is sort of the same idea.

When people hit <ctrl-a> to select all jobs, it's kinda like using try -u all (but not really). But e.g, if I typed 'mochitest-browser, then hit <ctrl-a><enter>, that would schedule code coverage jobs. This bug is proposing to require --full to do that.

Comment 3

[Greg Mierzwinski [:sparky]]

:ahal, do you think we could append all queries with '!pgo !ccov' to prevent those from running when '--full' is supplied or would that cause issues in the queries?

Comment 4

[Greg Mierzwinski [:sparky]]

(In reply to Greg Mierzwinski [:sparky] from comment #3)
> :ahal, do you think we could append all queries with '!pgo !ccov' to prevent
> those from running when '--full' is supplied or would that cause issues in
> the queries?

I mean when '--full' is not supplied.

Comment 5

[Andrew Halberstadt [:ahal]]

It's definitely possible. I think I'd prefer the solution outlined in comment 0 though. While it may be a bit hacky, I think it'll lead to a better user experience (plus it'll be generic, not just for |mach try fuzzy|).

Actually, if we fix both this bug generically and fix bug 1400295 (which I'd like to do sometime soonish) then bug 1473048 would magically be fixed for free.

Comment 6

[Greg Mierzwinski [:sparky]]

Oh, sounds good! That would be nice. :)

Comment 7

[Andrew Halberstadt [:ahal]]

Minor correction:
Bug 1473048 would be fixed for free when using `mach try`. It wouldn't be fixed for people using `hg push try`. So it's probably still worth fixing.

Comment 8

[Andrew Halberstadt [:ahal]]

Marco, do you agree that 'ccov tasks should be excluded from the default |mach try fuzzy| task list (and require --full to select them)?

Comment 9

[Andrew Halberstadt [:ahal]]

I think I'm going to ignore pgo for now, as whether or not those should be included in the default task set is a bit less cut and dry.

Comment 10

[Andrew Halberstadt [:ahal]]

Attachment: Bug 1397722 - [tryselect] Filter 'ccov' tasks out of the target task set (now they must be selected with --full), r=sparky

Usually people don't mean to schedule code coverage tasks on try. But e.g |mach
try fuzzy -q "'mochitest"|, will cause them to be scheduled as a side effect.
This results in wasted resources and superfluous data in ActiveData.

This patch makes it so you need to explicitly pass --full to select ccov tasks
(even though they're technically part of the target task graph).

Comment 11

[Greg Mierzwinski [:sparky]]

Comment on attachment 9002905 [details]
Bug 1397722 - [tryselect] Filter 'ccov' tasks out of the target task set (now they must be selected with --full), r=sparky

Greg Mierzwinski [:sparky] has approved the revision.

Comment 12

[Marco Castelluccio [:marco]]

(In reply to Andrew Halberstadt [:ahal] from comment #8)
> Marco, do you agree that 'ccov tasks should be excluded from the default
> |mach try fuzzy| task list (and require --full to select them)?

Sounds good to me. I don't like the fact that "mach try fuzzy" + ctrl-a could still cause backouts because what we run on mozilla-central would not be the same as what ctrl-a selects, but I guess it's OK in order to save resources.

Comment 13

[Andrew Halberstadt [:ahal]]

(In reply to Marco Castelluccio [:marco] (PTO until August 24) from comment #12)
> Sounds good to me. I don't like the fact that "mach try fuzzy" + ctrl-a
> could still cause backouts because what we run on mozilla-central would not
> be the same as what ctrl-a selects, but I guess it's OK in order to save
> resources.

Yeah, that's why I've been punting on this bug for so long. But on the other hand we want to discourage the use of a blank 'ctrl-a' anyway as it's wasteful. So I guess this is better.

Maybe I could add a "--no-filter" flag so you could still get the exact set of tasks needed to be 100% sure of not being backed out (might be useful in extreme situations). I guess in practice this isn't going to be a problem, and if it is we can tackle it in a follow-up.

Comment 14

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/dcedddab6d89
[tryselect] Filter 'ccov' tasks out of the target task set (now they must be selected with --full), r=sparky

Comment 15

[Andreea Pavel [:apavel]]

https://hg.mozilla.org/mozilla-central/rev/dcedddab6d89