851061 - Ability to opt into bustage notifications (eg email) for job types not shown in the default view

Status: RESOLVED WONTFIX

(Tree Management Graveyard :: TBPL, defect)

Description

[Ed Morley [:emorley]]

This is more of a reminder bug for when we work on the treeherder service/UI, since we won't be implementing this for TBPL-current.

tl;dr:
* We're always going to have job types that shouldn't be shown in the default view (eg not tier 1, not run frequently enough etc).
* Maintainers of those job types seem to dislike having to use &showall=1&jobname=theirJob in an apptab all the time, which results in things like xulrunner being broken for weeks before anyone notices + a lot of pushback from people when you say their job is ineligible for being unhidden.
-> Potential solution:  Ability to opt into notifications (eg email) for when job type X is broken for > N hours. 

(Not sure whether this would be built into treeherder-service, treeherder-UI or an additional downstream consumer of treeherder-service).

Full transcript from #releng:

{
bhearsum: edmorley: out of curiosity, was there any talk about how to deal with lack of visibility into nightly-only jobs at the recent work week?
edmorley: bhearsum: treeherder (ie tbpl2) will have better options for visibility groups
edmorley: ie more states than just visible/hidden, thus teams who have a vested interest in a build type can have them visible
...
edmorley: bhearsum: I large part of the problem is that nightly only = too large a regression range for sheriffs to be expected to deal with
edmorley: bhearsum: so in these cases, people's "hey I want my build unhidden" basically means "hey I want X to be green but I want to pawn the work off onto someone else; someone who doesn't have the specific knowledge to actually resolve any problems"
...
bhearsum: it's certainly not the biggest issue we have, but it's really frustrating for everyone when jobs break and they don't get noticed
bhearsum: my best example here is xulrunner -- sometimes it will break and go unnoticed for weeks, and then when we do notice (usually at a beta1), it's even harder to debug
edmorley: bhearsum: true re xulrunner - but the root cause isn't so much that it's hidden, it's that it doesn't have an owner that uses &showall=1 and jobname=xulrunner
edmorley: bhearsum: or perhaps we need to have email notifications for jobs like it?
bhearsum: yeah, maybe
bhearsum: i was talking about this with mbrubeck yesterday too
bhearsum: it doesn't always make sense for there to be an owner in these situations. eg, b2g where devs are focused on other branches, but trunk needs to not break too
bhearsum: and in both of those cases, it doesn't make sense to close the tree for bustage
bhearsum: they can be fixed out-of-band
edmorley: in those instances an async form of notification seems more appropriate then
edmorley: something like "for job types XYZ, don't show in the TBPL default view, but send an email notification to ABC if the last green jobs was more than X hours ago" (would avoid most of the false positives from all platform bustage)
bhearsum: ah, that's a good point
bhearsum: you don't want to be spammy if someone pushed an awful patch
edmorley: it would also catch the "builds have stopped being generated" case
bhearsum: edmorley: heh, we used to have nagios freshness checks on FTP...
bhearsum: edmorley: do you think e-mail (or similar) notifications are in the scope of treeherder?
edmorley: bhearsum: treeherder is going to be made up of a consumable web service as well as distinctly separate UI part, so a downstream consumer could poll the web service
edmorley: but worth discussing (when we get to that stage), whether it's worth incorporating into the service
bhearsum: i'm going to file a bug on reviving ftp-based freshness checks (or equivalent)
}

Comment 1

[Ed Morley [:emorley]]

> bhearsum: i'm going to file a bug on reviving ftp-based freshness checks

Bug 851058.

Comment 2

[Ed Morley [:emorley]]

Bug 1129647, bug 1123814 filed for Treeherder.