630538 - [Tracking bug] Stop using tinderbox server for Firefox status

Status: RESOLVED FIXED

(Release Engineering :: General, defect, P3)

Description

[Chris Cooper [:coop] (he/him)]

This is the tracking bug for what we've informally been calling "killing off tinderbox." 

I should note upfront that we are only trying to discontinue tinderbox server usage for Firefox. Other projects that are currently submitting data to the Mozilla tinderbox server can continue to do so, although you may wish to consider also moving to to TBPL or similar. We're not turning tinderbox server off, we're just not going to be sending Firefox data there any more.

At the recent Mozilla all-hands, we had a session on how to go about killing off
tinderbox server for good. Here's the list of blockers we came up with at the meeting:

1) Retain builds and logs for 30 days: bug 614786
2) Post logs to stage(ftp) for mobile and l10n builds
 a) mobile: bug 591168
 b) l10n: bug 625189
3) Error parsing for logs: bug 625887
4) Host tree status outside of tinderbox: bug 630534 
5) Pull comments from metrics db: bug 601743
6) Maintain a list of hidden builders: bug 630537

Comment 2

[Phil Ringnalda (:philor)]

This might block bug 621304, but it doesn't depend on it.

Comment 3

[Phil Ringnalda (:philor)]

Some annotating, since we sort of took over this bug to also track when we can stop sending email to Tinderbox for the Firefox-related trees:

bug 630534 doesn't actually block that - we can live with status living on Tinderbox while mail doesn't go there, we're already doing that for Try

bug 676879 blocks because even though the Valgrind builds on mozilla-central barely exist now, without mail to Tinderbox they would completely cease to exist

bug 685299 blocks because of the way the SpiderMonkey builds on mozilla-inbound currently only show a status, without having any way to get to a summary of why they failed or a log of the failure (and the same for Valgrind, once we start showing the status for them)

bug 686617 blocks both because having summaries in tbpl is how we tell what happened, and because not loading the summary means not having the "analyze the leak" links which are needed to recognize several leaks, even if you give up on the summary and just look a the log

Comment 4

[Armen [:armenzg]]

The mechanism through tinderbox's admin page to mark builders visible has been lost (see bug 687021). Marking it as a blocker.

Could we document what is the new way?

Comment 5

[John O'Duinn [:joduinn] (please use "needinfo?" flag)]

(In reply to Phil Ringnalda (:philor) from comment #3)
> Some annotating, since we sort of took over this bug to also track when we
> can stop sending email to Tinderbox for the Firefox-related trees:
> 
> bug 630534 doesn't actually block that - we can live with status living on
> Tinderbox while mail doesn't go there, we're already doing that for Try
Agreed.


> bug 676879 blocks because even though the Valgrind builds on mozilla-central
> barely exist now, without mail to Tinderbox they would completely cease to
> exist
> 
> bug 685299 blocks because of the way the SpiderMonkey builds on
> mozilla-inbound currently only show a status, without having any way to get
> to a summary of why they failed or a log of the failure (and the same for
> Valgrind, once we start showing the status for them)
per triage meeting: We can turn off sending emails from buildbot to tinderbox server for opt/debug builds, unittests, talos while still continuing to send emails from buildbot to tinderbox server for valgrind, ionmonkey, spidermonkey and fuzzer jobs. This will reduce load on tinderbox server immediately, which means faster processing for the remaining emails that are still being sent to tinderbox server. This also means we are no longer blocked on bug#676879 and bug#685299.


> bug 686617 blocks both because having summaries in tbpl is how we tell what
> happened, and because not loading the summary means not having the "analyze
> the leak" links which are needed to recognize several leaks, even if you
> give up on the summary and just look a the log
This is now fixed.

Comment 6

[Chris Cooper [:coop] (he/him)]

Attachment: Stop sending tinderbox mail by default.

The exceptions, i.e. projects/branches that will continue to send tinderbox mail, are:

* fuzzing
* ionmonkey
* nanojit
* spidermonkey
* valgrind

I've also disabled tinderbox mail for releases.

Comment 7

[Chris Cooper [:coop] (he/him)]

Comment on attachment 562449 [details] [diff] [review]
Stop sending tinderbox mail by default.

https://hg.mozilla.org/build/buildbot-configs/rev/278fd4f090d8

Comment 8

[Aki Sasaki (not active)]

This landed on production today.

Comment 9

[Aki Sasaki (not active)]

Hm, tracking bug tracking a single open bug (bug 685299), blocking a second tracking bug (bug 625979) that seems to be tracking the same issue.

Should we make bug 685299 block bug 625979, and resolve this bug?

Comment 10

[Chris AtLee [:catlee]]

Yes!