614943 - release e-mail improvements

Status: RESOLVED FIXED

(Release Engineering :: General, defect)

Description

[[e-mail] (:bhearsum)]

Using this bug to track various improvements to the automatic release mail.

To start with, staging mail should be easily distinguishable from production mail, probably by the subject line.

Comment 1

[Armen [:armenzg]]

I was thinking if we could put all emails wrt a release into the same thread? (Instead of sending each in a new thread)

I know in rel-drivers we have not been doing that way (we send one separate email for each releng action) but we might want to keep in one thread since it is now obvious when a releng (person) is replying to a thread and when the release automation is emailing (release@).

Do we have a list of builders that we DON'T want to email the state to rel-drivers? or do we want to send all?

Another improvement I thought is if we could send only one email when _all_ l10n builders are done instead of one per L10n builder.

Comment 2

[Syed Albiz [:salbiz]]

Attachment: add a tag to subject line to differentiate staging

Tested in staging, with a sendchange and a bunch of dummy steps.
Subject lines end up like: [staging-release] Tagging started for Firefox 3.6.11 build3

Unfortunately, this will require a reconfig, as interpolating the staging variable into the message templates doesn't really satisfy this requirement

Comment 3

[Syed Albiz [:salbiz]]

(In reply to comment #1)
> Do we have a list of builders that we DON'T want to email the state to
> rel-drivers? or do we want to send all?
> 

Is there anything specific you want not showing up? We have two pools of recipients for notification, the general group which gets everything(release@), and a special group (which for now, is also just release@)
which gets a bunch of specific steps, and only if they pass, (This is the one that will probably be switched over to r-d in the near future).

Comment 4

[Armen [:armenzg]]

I don't think they are interested on the "source" builder and/or partner repacks. I am not sure how interested they will be about the verification ones. We might want to ask them once we are closer to enable it on production.

Comment 5

[Chris AtLee [:catlee]]

Comment on attachment 493415 [details] [diff] [review]
add a tag to subject line to differentiate staging

Are you going to adjust the templates to get rid of the newline that appears at the beginning of the notification messages as well?

Comment 6

[Syed Albiz [:salbiz]]

Attachment: delete leading newline

Done, since this patch only touches the templates, it can be landed safely without a reconfig.

Comment 7

[[e-mail] (:bhearsum)]

Comment on attachment 493415 [details] [diff] [review]
add a tag to subject line to differentiate staging

Can you name the variable something other than "tag"? In the context of releases, it's confusing.

Comment 8

[Syed Albiz [:salbiz]]

Attachment: use 'message_tag' rather than 'tag' to avoid confusion

Done.

Comment 9

[Syed Albiz [:salbiz]]

Attachment: fix the platform check in the message code

Rail pointed out in IRC that the current method of determining the platform for messages was broken. I have attached a fix here.

Comment 10

[Rail Aliiev [:rail]]

Comment on attachment 493820 [details] [diff] [review]
fix the platform check in the message code

looks good

Comment 11

[Chris AtLee [:catlee]]

Comment on attachment 493820 [details] [diff] [review]
fix the platform check in the message code

Please add a comment explaining what this is doing...it took me several minutes to understand what you're trying to do!

Comment 12

[Syed Albiz [:salbiz]]

Attachment: add comment to createReleaseMessage for clarity

Done. Tests out in staging.

Comment 13

[[e-mail] (:bhearsum)]

Planning to do a reconfig with this tomorrow morning.

Comment 14

[[e-mail] (:bhearsum)]

Comment on attachment 493420 [details] [diff] [review]
delete leading newline

changeset:   1236:3a8a344f406f

Comment 15

[[e-mail] (:bhearsum)]

Masters have been updated.

Comment 16

[Syed Albiz [:salbiz]]

Attachment: Send an 'all repacks complete' email when all l10n chunks finish.

Uses a Dummy Builder fired off a dependent on the repack scheduler. In this patch, that email is only sent to the PassRecipients  bucket (in future r-d@), with all chunk status emails still heading through to release@. Tested on staging. We may want to craft a custom message template for this step.

Comment 17

[Chris AtLee [:catlee]]

Comment on attachment 496330 [details] [diff] [review]
Send an 'all repacks complete' email when all l10n chunks finish.

Do we want this per-platform, or across all platforms?

Comment 18

[[e-mail] (:bhearsum)]

Comment on attachment 496330 [details] [diff] [review]
Send an 'all repacks complete' email when all l10n chunks finish.

This seems OK to me, and I don't know of any other way to accurately fire a message when a set of repacks finish. Dustin, I'd be curious to know if you have any thoughts here.

Comment 19

[Dustin J. Mitchell [:dustin] (he/him)]

Comment on attachment 496330 [details] [diff] [review]
Send an 'all repacks complete' email when all l10n chunks finish.

Nope, this is the best we've got until we implement Flocks.  Well, the other option is to send the mail in a buildstep, but I think Syed's solution is better.

Comment 20

[[e-mail] (:bhearsum)]

Comment on attachment 496330 [details] [diff] [review]
Send an 'all repacks complete' email when all l10n chunks finish.

Landed on default: changeset:   1318:6d858c31b9bb

Comment 21

[Syed Albiz [:salbiz]]

Attachment: drop unnecessary mail to r-d

tested these on my latest staging run

Comment 22

[Syed Albiz [:salbiz]]

Attachment: add platform to repack emails, add updates template

individual chunks now go: %release: completed repack_x/y on %platform
all-finished now go: %release: completed l10n repacks on %platform
updates now go: %release: updates available on betatest

Comment 23

[[e-mail] (:bhearsum)]

Comment on attachment 499336 [details] [diff] [review]
add platform to repack emails, add updates template

Per IRC, we're going to symlink most of these to master copies.

Comment 24

[Syed Albiz [:salbiz]]

Attachment: link templates

Done.

Comment 25

[[e-mail] (:bhearsum)]

Comment on attachment 499357 [details] [diff] [review]
link templates

I think you typoed "macosx", looks fine otherwise.

Comment 26

[Syed Albiz [:salbiz]]

Attachment: fix typos

d'oh, old templates stuck around after I rsync-ed the templates dir in staging.

Comment 27

[[e-mail] (:bhearsum)]

Comment on attachment 499378 [details] [diff] [review]
fix typos

Landed on default: changeset:   1321:3c103d673c4f

Comment 28

[[e-mail] (:bhearsum)]

Comment on attachment 499334 [details] [diff] [review]
drop unnecessary mail to r-d

I landed this yesterday.

Comment 29

[[e-mail] (:bhearsum)]

From e-mail: we need to send links to the builds as they become available.

Comment 30

[Syed Albiz [:salbiz]]

Attachment: include ftpURL in emails

generate a generic ftp URL for the release and add platform specific information as available.

Comment 31

[Syed Albiz [:salbiz]]

Attachment: include ftpURL in templates

add ftpURL to builds, repacks and signing templates

Comment 32

[Rail Aliiev [:rail]]

Comment on attachment 500112 [details] [diff] [review]
include ftpURL in emails

>+    def genericFtpUrl():
>+        """ Generate an FTP URL pointing to the uploaded release builds for
>+        sticking into release notification messages """
>+        return makeCandidatesDir(
>+            releaseConfig['productName'],
>+            releaseConfig['version'],
>+            releaseConfig['buildNumber'],
>+            protocol='http',
>+            server=branchConfig['stage_server'])

IIRC, protocol should be 'ftp', because Apache has some rewrite rules, which prevent accessing the candidate directory directly. Not sure if they work for platform specific directories.

Comment 33

[[e-mail] (:bhearsum)]

Comment on attachment 500113 [details] [diff] [review]
include ftpURL in templates

>diff --git a/mozilla/release_templates/win32_build_success b/mozilla/release_templates/win32_build_success
>--- a/mozilla/release_templates/win32_build_success
>+++ b/mozilla/release_templates/win32_build_success
>@@ -1,4 +1,6 @@
>-%(releaseName)s: %(platform)s builds available
>-%(platform)s unsigned builds available for %(releaseName)s
>+%(releaseName)s: %(platform)s en-US build available

The subject here needs to say "unsigned" as well.

r=me with that changed.

Comment 34

[[e-mail] (:bhearsum)]

Comment on attachment 500112 [details] [diff] [review]
include ftpURL in emails

What does the FTP URL end up looking like post-signing? The root of the candidates directory?

As Rail suggested, we should use ftp:// as the protocol, to make sure the URLs are always accessible.

Comment 35

[Syed Albiz [:salbiz]]

Attachment: include ftpURL in templates-v2

Fixed subject line of the win32 build template

Comment 36

[Syed Albiz [:salbiz]]

Attachment: use ftp:// links

Done, using ftp:// URLs

Comment 37

[[e-mail] (:bhearsum)]

Comment on attachment 500190 [details] [diff] [review]
include ftpURL in templates-v2

changeset:   3716:efa343702b04

Comment 38

[[e-mail] (:bhearsum)]

Comment on attachment 500192 [details] [diff] [review]
use ftp:// links

changeset:   1334:ad2c3b1788f6

Comment 39

[[e-mail] (:bhearsum)]

The latest patches need to be merged to production still, but after that I believe we're all done here.

Comment 40

[[e-mail] (:bhearsum)]

These are all landed in production now.