1615618 - Ignore reviewbot try pushes to compute the percentile

Status: RESOLVED WONTFIX

(Release Engineering :: General, task)

Description

[Julian Descottes [:jdescottes]]

Follow up to a point I raised on Matrix' #treeherder channel:

Just noticed that pushing to try now shows duration estimates as well as the percentile of the push (which is great! thanks a lot for adding this). I was surprised that one of our presets ranked pretty badly (15% percentile, preset is "devtools-linux", takes for 11hours). We tried to make this preset as small as possible. So I looked at recent try pushes to see what others were using as try queries. The thing I noticed was that many try pushes were actually from reviewbot (eg https://treeherder.mozilla.org/#/jobs?repo=try&revision=60518bfc0737a9557093ef25b0b2b53f226c7e2d). And they are all super small by nature. I was wondering if we should try to exclude those from the calculation?

:armenzg suggested filing a bug and ni? :camd , so here it is!

Comment 1

[Cameron Dawson [:camd]]

Hi Julian-- I'm not sure which duration you're referring to, sorry! Would you attach a screenshot? Are you referring to when you hover over a task that's running and it shows how long it has been going? We don't have an overall estimate of how long the Push will take. Unless someone added it an I never noticed! :)

Comment 2

[Julian Descottes [:jdescottes]]

Hi Cameron!

I was referring to the information logged when pushing to try, in the command line:

estimates: Runs 97 tasks (85 selected, 12 dependencies)
estimates: Total task duration 1 day, 11:19:18
estimates: In the 19% percentile
estimates: Should take about 2:55:45 (Finished around 2020-02-19 11:07)

We don't have an overall estimate of how long the Push will take. Unless someone added it an I never noticed!

Looks like it :)

Also, I don't know for sure if reviewbot's pushes are counted (the percentiles have moved quite a bit since last week, for instance this 1day11hrs push used to be 10% percentile, now it's 19%). It was just a guess.

Comment 3

[Julian Descottes [:jdescottes]]

Looks like this was added by Bug 1611989, I should probably move the Bug there.

Comment 4

[Julian Descottes [:jdescottes]]

Hi Simon,

I see you implemented the initial change to show task duration when pushing to try. I was curious about which pool of try pushes was used to calculate the percentiles, and whether it included reviewbot's try pushes. Maybe you know?

Comment 5

[Simon Fraser [:sfraser] ⌚️GMT]

(In reply to Julian Descottes [:jdescottes] from comment #4)

the percentiles have moved quite a bit since last week, for instance this 1day11hrs push used to be 10% percentile, now it's 19%

Well spotted - I started filtering out things from try where the decision task failed, which was a surprising number, and it reduced the quantiles quite a lot due to the number of small durations produced.

I see you implemented the initial change to show task duration when pushing to try. I was curious about which pool of try pushes was used to calculate the percentiles, and whether it included reviewbot's try pushes. Maybe you know?

We're not filtering out reviewbot at the moment. Our current query is doing this:

  WHERE
    project = 'try'
    AND cost IS NOT NULL
    AND tasks > 1 -- Ignore things that are likely only a decision task, due to rate of failures there
    AND date > DATE_SUB(@run_date, INTERVAL 60 day) )

We don't currently have the 'owner' of those tasks bubble up to the task group summary table as there's often more than one: plenty of sheriff and release tasks involve adding more tasks to an existing task group. I count 2322 entries like that for things run on 'try' in the last 60 days.

I'll have a think about the best way to filter out this sort of automation

Comment 6

[Julian Descottes [:jdescottes]]

(clearing the ni for camd)

Thanks for the feedback Simon! I really appreciate having the metrics, happy to see that we are still tuning things out. Feel free to keep the bug open if it's something worth tracking on its own, or to close it if you are already tracking this somewhere else.

Comment 7

[Simon Fraser [:sfraser] ⌚️GMT]

I'll keep this bug open, if that's ok. I've an idea on how to get the owners included, and we can then filter out reviewbot and github addresses (as they reach try for CI tasks as well)

Comment 8

[Cameron Dawson [:camd]]

(In reply to Julian Descottes [:jdescottes] from comment #2)

Hi Cameron!

I was referring to the information logged when pushing to try, in the command line:

estimates: Runs 97 tasks (85 selected, 12 dependencies)
estimates: Total task duration 1 day, 11:19:18
estimates: In the 19% percentile
estimates: Should take about 2:55:45 (Finished around 2020-02-19 11:07)

We don't have an overall estimate of how long the Push will take. Unless someone added it an I never noticed!

Looks like it :)

Oh, ok. So this isn't part of Treeherder, that's why I didn't know about it. :) This looks like it's in the right component now.

Also, I don't know for sure if reviewbot's pushes are counted (the percentiles have moved quite a bit since last week, for instance this 1day11hrs push used to be 10% percentile, now it's 19%). It was just a guess.

Comment 9

[BugBot [:suhaib / :marco]]

The bug assignee is inactive on Bugzilla, so the assignee is being reset.

Comment 10

[Julien Cristau [:jcristau]]

Bug 1977995 removed these estimates.