1421273 - Show count of jobs classified as "fixed by commit" for push

Status: RESOLVED FIXED

(Tree Management :: Treeherder: Frontend, defect, P3)

Description

[Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)]

When a sheriff looks for merge candidates, they have to check if a push contains a regression by looking for failure in that push marked with "fixed by commit". At the moment, this can be done by clicking on the filter button and selecting both "failure classification" and "fixed by commit" and clicking on "add".

It should be a number instead of a binary information because it is a hint for a person who classifies still unstarred jobs if it's likely that a failure is yet another occurrence of a regression which got fixed by a backout.

Comment 1

[Ed Morley [:emorley]]

(In reply to Sebastian Hengst [:aryx][:archaeopteryx] (needinfo on intermittent or backout) from comment #0)
> It should be a number instead of a binary information because it is a hint
> for a person who classifies still unstarred jobs if it's likely that a
> failure is yet another occurrence of a regression which got fixed by a
> backout.

If the whole push were marked in some way for "backed out by commit", would this achieve the same result?

Comment 2

[Phil Ringnalda (:philor)]

I'd say so, yes. The only numbers that matter are "0" and "more than 0", so this being marked as a duplicate of a reopened bug 1295679 seems reasonable.

Of course, it's tricksier than just "this push has fixed-by-commit so it is bad" since I star things as fixed-by-commit when the fixing commit is on mozilla-central and hasn't yet been merged around, and the push or ten pushes you are looking at may not have a fixed-by-commit job if it's a SETAed job and then there was build bustage when it would have run, which is why what I always wanted was UI something like a red background on the left column where the push is shown, linked to an improved https://treeherder.mozilla.org/#/jobs?repo=mozilla-inbound&filter-failure_classification_id=2 that has lines running from each marked job up to the commit it says fixed it (or running up off the top of the page for commits that don't yet exist). And a pony.

Comment 3

[Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)]

(In reply to Ed Morley [:emorley] from comment #1)
> (In reply to Sebastian Hengst [:aryx][:archaeopteryx] (needinfo on
> intermittent or backout) from comment #0)
> > It should be a number instead of a binary information because it is a hint
> > for a person who classifies still unstarred jobs if it's likely that a
> > failure is yet another occurrence of a regression which got fixed by a
> > backout.
> 
> If the whole push were marked in some way for "backed out by commit", would
> this achieve the same result?
No, see the second abstract of comment 0.
1. This is not only about the push which require backout but the ones between the former landing and getting backed out.
2. It's also about caching all the perma fail instances, e.g. a bc test fails always on Linux, then X (reduced test set) or Y (full test set) jobs should be marked as fixed by commit (with multiple sheriffs per shift, the information might not be in the brain of the sheriff currently doing the classification).

Comment 5

[Cameron Dawson [:camd]]

Fixing this will involve modifying the code here:

https://github.com/mozilla/treeherder/blob/7f72192565a6eac8af503de67a2783a9756a20a3/ui/job-view/pushes/Push.jsx#L73

You would want to count any job that has failure_classification_id of 2 (which is "Fixed by Commit" here: https://github.com/mozilla/treeherder/blob/4cfdae3f41d656840486b771501f6498836e7710/treeherder/model/fixtures/failure_classification.json#L10)

Next, you want to display that new value. That should be done here: https://github.com/mozilla/treeherder/blob/7f72192565a6eac8af503de67a2783a9756a20a3/ui/job-view/pushes/PushHeader.jsx#L419

If the value is above 0, then it should be shown in red. Use your discretion on how to make it look. Perhaps something that would stand out to a user with color vision impairment.

Comment 6

[Cameron Dawson [:camd]]

In addition, when you've got your changes in a state where you can start testing, you can run your code against our stage environment. You start the code in your command-line with yarn start:stage. You should be able to classify a job as "Fixed by commit", once you're logged in.

When you classify it, your count should update on the top-row of the push.

Comment 7

[Cameron Dawson [:camd]]

Attachment: Screenshot 2019-10-07 10.56.47.png

Comment 8

[Cameron Dawson [:camd]]

See attachment for where these counts ( and your new count ) will display on each push.

Comment 9

[GitHub Bugzilla PR Linker]

Attachment: Link to GitHub pull-request: https://github.com/mozilla/treeherder/pull/5536

Comment 10

[GitHub Bugzilla PR Linker]

Attachment: Link to GitHub pull-request: https://github.com/mozilla/treeherder/pull/5537

Comment 11

[Cameron Dawson [:camd]]

Nice work, Pachi!