1216602 - Exclude jobs that rely on external resources from War on Orange report

Status: RESOLVED WONTFIX

(Tree Management Graveyard :: OrangeFactor, defect)

Description

[Maja Frydrychowicz :impossibus (was :maja_zf) (needinfo me)]

From latest report:
>Trunk orange factor: 14.6 (9915 oranges in 678 pushes)
>Last week's trunk orange factor: 12.1 (7228 oranges in 595 pushes)
>This week's top ten oranges in trunk:
> 1. 526 oranges: bug 1211959 (NEW)
>    <https://bugzilla.mozilla.org/show_bug.cgi?id=1211959>
>    Intermittent media tests  FAIL test_video_playback.py
>    TestVideoPlayback.test_playback_starts | after  AssertionError:
>    TimeoutException: Timed out after 62.8 seconds; condition:
>    playback_started

media-tests jobs (group VP in Treeherder) are Tier-2. They will always have a relatively high number of intermittent failures due to interacting directly with YouTube and other video sites. Does it make sense to include these failures in Orange Factor analysis?

Some discussion today: 
<nigelb> On the one hand, tier 2 is useful for jobs that are flaky but need to be unflaky at some point.
<nigelb> But these jobs are always going to be flaky, like the push tests
<nigelb> so somes sort of blacklist would be nice.
<nigelb> "We don't count this bug/failure for automatic war on orange counts" or some such.
<nigelb> maja_zf: We've had a discussion on the sheriffs list about how to get TH to make tier 2 sort of out of the way of day-to-day sheriffing. (bug 1148674)

Comment 1

[Maja Frydrychowicz :impossibus (was :maja_zf) (needinfo me)]

Since Orange Factor may be going away in favour of an implementation in Treeherder, seeking input from Treeherder team.

Comment 2

[Ed Morley [:emorley]]

At the moment OrangeFactor doesn't know which are tier 2 jobs, so there are a few options:
1) Send the tier to ElasticSearch, along with the other job properties, and filter it out of the weekly emails (or display them in their own section in the emails etc)
2) Filter by tier in Treeherder and just don't sync those classifications to ElasticSearch at all

Though given that we'll be rewriting OrangeFactor from scratch, and that this is just an email send to auto-tools/dev.tree-management, I wonder if it's worth spending the effort handling this for now?

Comment 3

[Ed Morley [:emorley]]

Reading what nigelb wrote in IRC from comment 0, I think filtering out tier 2 entirely might not be ideal, so some kind of blacklist would be good - but I guess that might be better done using the treeherder job signatures, rather than the suboptimal fields available in ElasticSearch at present (and thus another reason to wait for the rewrite of OF).

Comment 4

[Ed Morley [:emorley]]

OrangeFactor is now EOL (bug 1367364), having been replaced by the new Intermittent Failures View (in bug 1367362):
https://treeherder.mozilla.org/intermittent-failures.html

Bugs for the new view can be filed here:
https://bugzilla.mozilla.org/enter_bug.cgi?product=Tree%20Management&component=Intermittent%20Failures%20View