678064 - Create WOO view for tracking test failures that were never starred

Status: RESOLVED WONTFIX

(Tree Management Graveyard :: OrangeFactor, defect)

Description

[Jonathan Griffin (:jgriffin)]

When looking through test failures for bug 646749, I discovered several cases where test failures on mozilla-central were never starred.  One cause was that mochitest is missing some failures in its summary, filed as bug 677964.  But there are other causes as well.

It would be easy to take the data from WOO and display a list of test failures that had no TBPL comments; presumably for mozilla-central this list should be zero.  Any failures appearing in such a list would be candidates for investigation of TBPL/buildbot/test harness problems.

To get the best matches for this, we'd need a version of the /testfailures API that matched TBPL comments (/tbpl/comments in ElasticSearch) against test failures, instead of just bugs as it presently does.

Comment 1

[Jonathan Griffin (:jgriffin)]

Attachment: server API to retrieve list of unstarred failures

Comment 2

[Mark Côté [:mcote]]

Comment on attachment 557364 [details] [diff] [review]
server API to retrieve list of unstarred failures

Cool, I verified that indeed some of the oranges returned by this patch show up on tbpl as unstarred.

I've found a few unstarred oranges being ignored by your patch, although they are either Talos, Android, or Jetpack.  Is this intentional?  I think we are ignoring the first two in WOO, but aren't we paying attention to Jetpack?  For example, https://tbpl.mozilla.org/?noignore=1&rev=1c2522e75c04 shows an unstarred Jetpack orange in Win debug, but your patch only returned the unstarred WinXP opt xpcshell for that revision. Just wanted to make sure you're aware.  But maybe this had to do with the MPT outage?

Comment 3

[Jonathan Griffin (:jgriffin)]

(In reply to Mark Côté ( :mcote ) from comment #2)
> Comment on attachment 557364 [details] [diff] [review]
> server API to retrieve list of unstarred failures
> 
> Cool, I verified that indeed some of the oranges returned by this patch show
> up on tbpl as unstarred.
> 
> I've found a few unstarred oranges being ignored by your patch, although
> they are either Talos, Android, or Jetpack.  Is this intentional?  I think
> we are ignoring the first two in WOO, but aren't we paying attention to
> Jetpack?  For example, https://tbpl.mozilla.org/?noignore=1&rev=1c2522e75c04
> shows an unstarred Jetpack orange in Win debug, but your patch only returned
> the unstarred WinXP opt xpcshell for that revision. Just wanted to make sure
> you're aware.  But maybe this had to do with the MPT outage?

This is actually caused by the fact that the logparser doesn't know how to parse jetpack logfiles; bug 684274 just filed.  This won't affect the OF itself, however, just functionality, like this API, that utilizes actual error statements from the log.

Comment 4

[Jonathan Griffin (:jgriffin)]

server patch landed as http://hg.mozilla.org/automation/orangefactor/rev/a4f4fe372a79

Comment 6

[Ed Morley [:emorley]]

OrangeFactor is now EOL (bug 1367364), having been replaced by the new Intermittent Failures View (in bug 1367362):
https://treeherder.mozilla.org/intermittent-failures.html

Bugs for the new view can be filed here:
https://bugzilla.mozilla.org/enter_bug.cgi?product=Tree%20Management&component=Intermittent%20Failures%20View