1328610 - SETA - only pick PGO test failures if they're truly the root cause

Status: RESOLVED WONTFIX

(Tree Management Graveyard :: Treeherder: SETA, defect)

Description

[Armen [:armenzg]]

Attachment: Output from http://localhost:8000/api/seta/failures-fixed-by-commit/

I have noticed this in the past. I don't think it is a regression but I believe it could be an issue all the way in ouija (since I barely modified the high_value_jobs module; named differently in ouija).

Given the failures fixed by commit according to the attachment, we can see that over 10 jobs were fixed by commit 0496c868606e.

With this limited data, I would consider all jobs as high values, however, analyze_failures determines that only one job is to be considered high value [1]

We can easily make a test case based on this output to test the logic from the high_value_jobs.py module [2] (formerly known as seta.py).

[1]
[2017-01-04 14:52:39,756] INFO [treeherder.seta.high_value_jobs:63] jobtype: (u'mochitest-browser-chrome-4', u'pgo', u'windows7-32-vm') is the root failure(s) of these [u'0496c868606e'] revisions
[2] https://github.com/mozilla/treeherder/blob/8b42a388661a472ca203f86f0546788442f2cb23/treeherder/seta/high_value_jobs.py
[3] https://github.com/mozilla/ouija/blob/c66a68d888661a6f480f75032ccc20e708f4d5cc/tools/seta.py#L65-L225

Comment 1

[Armen [:armenzg]]

Also notice that it marked a PGO test job as high value:
"Windows 7 VM 32-bit mozilla-inbound pgo test mochitest-browser-chrome-4"

Comment 2

[Armen [:armenzg]]

It seems that the behaviour on the description is correct, however, comment 1 shows that we should have not picked up a PGO test jobs but one of the other ones.

We might be able to fix this by sorting the list of jobtypes for a revision and putting the PGO test jobs at the beginning of the list. That could make it so we pick pgo test jobs as the last option for determining a root cause.

Comment 3

[Armen [:armenzg]]

Perhaps here:
https://github.com/mozilla/treeherder/blob/8b42a388661a472ca203f86f0546788442f2cb23/treeherder/seta/high_value_jobs.py#L113