633657 - automatic email to mirrors of pending releases

Status: RESOLVED WONTFIX

(Release Engineering :: Release Automation: Other, defect, P5)

Description

[John O'Duinn [:joduinn] (please use "needinfo?" flag)]

From discussion with justdave:

1) We need to forewarn mirrors of upcoming releases. Warning mirrors *at the same time* as we start pushing to mirrors is not ok. Typically justdave gave 24-48 hours notice, but he is also happy with 72 hours or above.

2) To "automate" this, RelEng will trigger buildbot to send email once we get QA signoff on a release. In later enhancement, if we change how QA does signoff, the QA signoff could be detected by buildbot, and trigger sending the email without human intervention. 

3) here's a boilerplate example of the email we need to send. Nothing set in stone here, so tweak if it helps.

=====

-------- Original Message --------
Subject: Firefox releases coming to mirrors
Date: Fri, 11 Feb 2011 16:09:04 -0800
From: [e-mail]
To: [e-mail]


On Monday (14th), we plan to "push to mirrors" for the Firefox 3.6.14 and Firefox 3.5.17 security releases.

If you believe this will cause any problems, or there are other releases
scheduled for those dates on mirrors, please let us know at
[e-mail].

Thanks
John.

=====

Comment 1

[Chris AtLee [:catlee]]

Recommend INCOMPLETE or WONTFIX. Adding a new manual step in our automation seems like a bad idea.

Release-drivers should be communicating planned release times to mirrors.

Comment 2

[John Ford [:jhford] CET/CEST Berlin Time]

There is also the case for a chem spill where we might want to warn the mirrors as soon as we have a fix because they are short turnaround.

Comment 3

[Mike Taylor [:bear]]

Justdave,

To be clear this would cause the RelEng scripts to send the email notification automatically once the post-QA step is automated.

Is this what your intent is?  To completely remove the manual step?

Comment 4

[John O'Duinn [:joduinn] (please use "needinfo?" flag)]

(In reply to comment #0)
> From discussion with justdave:
> If you believe this will cause any problems, or there are other releases
> scheduled for those dates on mirrors, please let us know at
> [e-mail].

If you believe this release will cause any problems, please notify [e-mail]. If you have any problems with mirror status/weights/etc, please notify [e-mail].

(In reply to comment #1)
(In reply to comment #2)
(In reply to comment #3)
Working out details on how precise we need to be in our mirror notifications. 

For now, we are taking the manual process which used to be done by justdave, and having it instead be done manually by the release engineer for that specific release. This avoids missed release announcements, and helps us formalize what we send to mirrors. Once we have this data we can decide if this is automate-able, or should remain a manual task and be WONTFIXd.

Comment 5

[Dave Miller [:justdave]]

Here's an example of the type of message we typically sent by hand:

---8<---
Date: Mon, 18 Oct 2010 18:47:00 -0400
From: Dave Miller <[e-mail]>
To: [e-mail]
Subject: Multiple Firefox, Thunderbird, and SeaMonkey releases Tues Oct 19

Firefox 3.6.11 and 3.5.14; Thunderbird 3.1.5 and 3.0.9; and SeaMonkey 
2.0.9 will all be releasing on Tuesday October 19, as a regularly 
scheduled security update to Firefox, as per 
https://wiki.mozilla.org/Releases .

The bits should start getting pushed out to the mirrors sometime in the 
early morning PST (-0800) Tuesday morning, with the release 
announcements coming in the late morning or early afternoon.

Being that this release is a security update getting pushed through the 
automatic update service, the vast majority of the userbase will be 
downloading within the first 24 hours after release.  It will be 
combined traffic of both Firefox 3.6.x and 3.5.x users, as well as the 
ever-increasing pool of testers of the 4.0 betas, plus SeaMonkey and 
Thunderbird, so this will be a full-traffic release.  SeaMonkey traffic 
will only be sent to those of you mirroring the mozilla-releases module 
(mozilla-current doesn't contain SeaMonkey).

If you get too much traffic, please let us know and we can adjust your 
weights in the download redirector.  You should be able to find a 
sysadmin most of the time in #mirrors on irc.mozilla.org.  Feel free to 
stop in any time if you have questions about anything as well.

As well, if you think you can handle more traffic than you're getting, 
let us know that, too (but keep in mind that you'll normally get at 
least 10 times the normal traffic on a release day).  You can see your 
current weight at https://nagios.mozilla.org/sentry/ (use 'guest' as 
both the username and password).

As usual, if this presents any problems or anyone knows of any 
conflicting releases with other projects hosted on your mirrors, let us 
know!
---8<---

Note that there's usually a little bit of customization around the nature of the release and which products are included, which may or may not be possible with an automated notification, but most of the latter part of that email was the same template every time.

Comment 6

[[e-mail] (:bhearsum)]

Mass move of bugs to Release Automation component.

Comment 7

[[e-mail] (:bhearsum)]

We don't send these mails anymore.