1063442 - Release runner should not spam release-drivers email address

Status: RESOLVED DUPLICATE of bug 1063408

(Release Engineering :: General, defect)

Description

[Pete Moore [:pmoore][:pete]]

When first trying to release esr31 fennec armv6 (31.1.0esr build 1 - bug 1040319), release runner failed (and spammed release-drivers every minute) due to it not having an updated checkout of buildbot-configs.

It should probably update buildbot-configs automatically before validating the release, and probably should not spam release-drivers in case of problems (e.g. maybe we could add a second email address field in release runner config for technical failures / python exceptions?).

Comment 1

[Pete Moore [:pmoore][:pete]]

Presumably also tools repo - I did a manual update on buildbot-master81, and it seemed quite out of date.

Comment 2

[Pete Moore [:pmoore][:pete]]

The first part of this bug was solved in bug 1067581 - leaving the outstanding part of changing distribution email for failures to go to release engineering team, rather than release drivers. Maybe one email should be sent to release drivers (e.g. the first?) - but for repeated failures, they should just go to [e-mail]?

Comment 3

[Nick Thomas [:nthomas] (UTC+12)]

OK. There's also bug 1063408, where I assert that failures shouldn't go to r-d at all.

Comment 4

[Pete Moore [:pmoore][:pete]]

Thanks Nick.

I think https://bugzilla.mozilla.org/show_bug.cgi?id=1042056#c0 still holds (i.e. release drivers should at least be notified that there is a problem if releng are not around) - so maybe we should look at finding a solution which balances the following principles:

  * release drivers should not be repeatedly spammed with technical lengthy output that might not be useful to them
  * release drivers should receive an email if there is a problem

Maybe we would have something along the lines of "There has been a problem making this release - logs can be downloaded from ....." going to release drivers, and any technical failures (especially repeated ones) get sent to [e-mail]. Maybe we keep a track of when a problem first occurs (rather than the retries) and only the first failure causes a release drivers email, and then if it succeeds, it could also send a release drivers email. The existing behaviour for the repeated emails could continue to spam release@ email address. What do you think about that? I'm trying to think of a way we can balance both of the above goals.

Pete

Comment 5

[Pete Moore [:pmoore][:pete]]

(the unstated third goal, of course, is that release@ email continues to get notifications of each and every failure)

Comment 6

[Nick Thomas [:nthomas] (UTC+12)]

There's actually a big difference between RelMan and release-drivers. The latter is a very wide group of people covering everyone from press/marketing to us. I don't think they need to know about errors, only when a release has actually started successfully (ie I disagree with when we notify at the moment.)

I'm not suggesting we suppress errors, or build a system for parsing them, just that we only send problems to us + relman.

Comment 7

[Pete Moore [:pmoore][:pete]]

Thanks Nick - that was a misunderstanding on my part - I thought emailing release-drivers was how we communicated to Release Management. So that makes perfect sense. We can spam RelEng and RelMan as much as need, and just send success notifications to Release Drivers. =)

I guess with the work Zeller has been doing to capture state in ship it, maybe we can send an email when we know the release is done and finished, rather than when it has been triggered?

Comment 8

[Nick Thomas [:nthomas] (UTC+12)]

Duping to earlier bug.