1698018 - Update defaults for custom backfill option

Status: NEW

(Tree Management :: Treeherder, enhancement, P2)

Description

[Ionuț Goldan [:igoldan]]

Whenever I backfill as a perf sheriff, I use the custom backfill option.
For every job I need to backfill, I have to manually change the inclusive: false to true and times: 1 to 5.

We should replace these 2 defaults to true and 5 respectively, to optimize the command for perf sheriffing.

Comment 1

[Ionuț Goldan [:igoldan]]

Attachment: change-custom-backfill-defaults.PNG

Comment 2

[Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)]

This would require the code sheriffs to revert the values for their backfills to revert these values to prevent unnecessary load.

A separate action for backfills by perf sheriffs could be added.

Two comments:

1. Requesting multiple runs per push causes duplicate builds (one for each task), see bug 1658174. This affects performance tasks because they mostly use shippable builds which are only build by default for pushes which also run perf tasks by default.
2. The combination of multiple runs per push + running it for every push causes significant load on the hardware pools. Could pushes unrelated to alerts be omitted?

Comment 3

[Ionuț Goldan [:igoldan]]

(In reply to Sebastian Hengst [:aryx] (needinfo on intermittent or backout) from comment #2)

[...]
2) The combination of multiple runs per push + running it for every push causes significant load on the hardware pools. Could pushes unrelated to alerts be omitted?

I don't understand what "pushes unrelated to alerts" means. If the suggestion is to implement an heuristic to somehow determine what to omit, that's definitely not something we can practically do. Perf regressions tend to have very unexpected culprits.

The load on hardware pools is something expected. It's a drawback from reducing the job frequency from every 5 pushes to every 20 pushes. There's 4 times more missing data. Thus, the load is 4 times bigger than it used to be.
Perf sheriffs need this (missing) data one way or the other. And we surely need it ASAP, because we have very clear & strict KPIs. If the load is a problem, more advanced Taskcluster mechanisms should be employed or the pool size should be increased.

Comment 4

[Joel Maher ( :jmaher ) (UTC -8)]

I would think a separate perf_backfill action could be defined.

This is an action that is has been in use for years and :igoldan is trying to reduce the manual steps a bit, so there is no worry about load on our existing hardware pools. I think the code sheriffs backfill more frequently than perf sheriffs which would make this default not ideal for code sheriffs, and more evidence we need a separate action.