Closed Bug 398494 Opened 12 years ago Closed 9 years ago

use automation, not humans, to send notification emails to other groups

Categories

(Release Engineering :: General, defect, P5)

defect

Tracking

(Not tracked)

RESOLVED FIXED

People

(Reporter: joduinn, Assigned: salbiz)

References

Details

(Whiteboard: [automation])

Attachments

(8 files, 9 obsolete files)

1.33 KB, patch
rhelmer
: review+
Details | Diff | Splinter Review
51 bytes, text/plain
Details
1.58 KB, patch
rhelmer
: review+
Details | Diff | Splinter Review
17.18 KB, text/plain
Details
1.60 KB, patch
rhelmer
: review+
Details | Diff | Splinter Review
10.59 KB, patch
rail
: review+
bhearsum
: checked-in+
Details | Diff | Splinter Review
10.33 KB, patch
bhearsum
: checked-in+
Details | Diff | Splinter Review
3.37 KB, patch
bhearsum
: review+
bhearsum
: checked-in+
Details | Diff | Splinter Review
Each major step in the build automation sends emails to build@mozilla.org, with terse status updates in the body. During the FF2.0.0.7 release, we found that we were forwarding each of these emails to release-drivers, adding some human friendly text for context.

It would be nice if:
- "all good" emails were changed to be more human friendly, and buildbot could be sent to release-drivers as well as build.
- "something wrong" emails continue to only be sent to build. People in the build group could then analyze what went wrong and sent a proper update to release-drivers.

Its possible that some "all good" emails should not be sent to release-drivers, but what we did for 2007 seems ok. This needs to be investigated.
For the 2.0.0.8rc1, 2.0.0.8rc2 releases, there was a few times where the automation finished and sent email to build@m.o... only to have a few hours pass before a human saw that email, and forwarded the email to release-drivers, with some descriptive text. Making these success emails nicer would eliminate this delay. 


As a starting point for discussions, I'd suggest the following:

- all error emails are sent to build@m.o. 

- all error emails be also sent to a new not-yet-created build-notify@m.o. list, so whomever is working that specific release could be SMS'd / paged for these errors?

- all success emails are sent to build@m.o.

- a subset of success emails be also sent to release-drivers.
-- tag started
-- linux build available
-- mac build available
-- signed win32 build available
-- update snippets available on betatest channel
-- update snippets available on beta channel
-- update snippets available on release channel
Priority: -- → P3
Assignee: build → nobody
QA Contact: mozpreed → build
Status: NEW → ASSIGNED
Priority: P3 → P2
Assignee: nobody → joduinn
Status: ASSIGNED → NEW
Turns out I dont have necessary permissions to create mailing list in mailman. Contacted admin to figure out how to get this.

Meanwhile, I noticed in the buildbot doc that buildbot.status.mail.MailNotifier has 3 different modes : 
    * 'all': send mail about all builds, passing and failing
    * 'failing': only send mail about builds which fail
    * 'problem': only send mail about a build which failed when the previous build passed
...which is fine for the error emails to build. However, for the success emails to release-drivers, we'll have to work something else out.
Further thoughts on doing "success-only" notification given BuildBot's MailNotifier 3 modes (all/failing/problem). 

We could create a new "Notify Release-Drivers" step, which would always notify release-drivers, and yet would only be called if the previous step was successful. This mimics the success-only behavior we want. If we did this, we'd need to add that new step here in these factories:
- tagFactory (both as first and as last step)
- buildFactory (as last step)
- signFactory (as last step)
- stageFactory (as last step)

The only downside I can see is that it means we would notify for unsigned win32 builds, as well as later for signed win32 builds. We had been trying to only notify on signed win32 builds, to save QA redoing manual work.
Status: NEW → ASSIGNED
(In reply to comment #2)
> ...which is fine for the error emails to build. However, for the success emails
> to release-drivers, we'll have to work something else out.
> 

I bet it would be _really_ easy to patch Buildbot to do this; and would probably be accepted upstream.

> We could create a new "Notify Release-Drivers" step, which would always notify
> release-drivers, and yet would only be called if the previous step was
> successful. This mimics the success-only behavior we want. If we did this, we'd
> need to add that new step here in these factories:
> - tagFactory (both as first and as last step)
> - buildFactory (as last step)
> - signFactory (as last step)
> - stageFactory (as last step)
>
> The only downside I can see is that it means we would notify for unsigned win32
> builds, as well as later for signed win32 builds. We had been trying to only
> notify on signed win32 builds, to save QA redoing manual work.

This would be simplified if we patched MailNotifier. We could just do something like...
c['status'].append(MailNotifier(
    builders=["tag", "linux_build", "macosx_build", "sign", "stage"],
    mode="success",
    ...
)

IMHO this is the preferred approach.
I was about to write a patch for this, but I found this on buildbot.net. I'm not sure about licensing/rights to this, but since it's posted on buildbot.net can we assume it's free enough? If not, I can rewrite.
Attachment #298446 - Flags: review?(rhelmer)
Comment on attachment 298446 [details] [diff] [review]
[checked in] support 'passing' mode in MailNotifier

Where is this from on buildbot.net, an outstanding patch in the bug tracker? We should make sure that it gets landed, or we'll have to re-merge this.

What version of buildbot is this a patch against?
http://buildbot.net/trac/ticket/169

I *think* the patch is for Buildbot 0.7.6, but I'm not 100% sure.

but I did a test apply against our repo on the BUILDBOT_MOZILLA_PRODUCTION tag and it applies cleanly + unit tests pass. It applies to the trunk of our CVS with a bit of help.
Comment on attachment 298446 [details] [diff] [review]
[checked in] support 'passing' mode in MailNotifier

Seems fine to me, we need some way of tracking our outstanding upstream bugs. Too bad launchpad isn't opensource :)
Attachment #298446 - Flags: review?(rhelmer) → review+
reassigning to bhearsum, as he found this new patch which looks like it should do the trick...
Assignee: joduinn → bhearsum
Status: ASSIGNED → NEW
I need to convert the BUILDBOT_MOZILLA_PRODUCTION tag to a branch in order to land this in the right place. This is what I intend to do:

cvs rtag -F -r BUILDBOT_MOZILLA_PROUDCTION -b BUILDBOT_MOZILLA_PRODUCTION mozilla/tools/buildbot

Can I get your r? on this, Rob?
Assignee: bhearsum → nobody
Sorry for the bugspam, i clicked the wrong checkbox.
Assignee: nobody → bhearsum
Status: NEW → ASSIGNED
(In reply to comment #10)
> I need to convert the BUILDBOT_MOZILLA_PRODUCTION tag to a branch in order to
> land this in the right place. This is what I intend to do:
> 
> cvs rtag -F -r BUILDBOT_MOZILLA_PROUDCTION -b BUILDBOT_MOZILLA_PRODUCTION
> mozilla/tools/buildbot
> 
> Can I get your r? on this, Rob?

Yeah, with s/BUILDBOT_MOZILLA_PROUDCTION/BUILDBOT_MOZILLA_PRODUCTION/ :)

I've never converted a tag to a branch like this before, usually tend to create a branch from a base tag so you know where the branch started, not sure it really matters in this case though.

Maybe it'd be clearer to have BUILDBOT_0_7_5_BRANCH, BUILDBOT_0_7_6_BRANCH, etc. and force-tag BUILDBOT_MOZILLA_PRODUCTION to the version we want? 

Either way is fine with me.
Okay, as per discussion on IRC I'm going to do the following:
cvs rtag -r BUILDBOT_MOZILLA_PRODUCTION -b BUILDBOT_0_7_5_BRANCH
and then land this patch there.

It would be _good_ if we could branch from BUILDBOT_0_7_5_RELEASE, but then we would have to re-land all of the commits since, and that's more effort than it is worth.
Comment on attachment 298446 [details] [diff] [review]
[checked in] support 'passing' mode in MailNotifier

Checking in buildbot/status/mail.py;
/cvsroot/mozilla/tools/buildbot/buildbot/status/mail.py,v  <--  mail.py
new revision: 1.1.1.1.2.1; previous revision: 1.1.1.1
done
Attachment #298446 - Attachment description: support 'passing' mode in MailNotifier → [checked in] support 'passing' mode in MailNotifier
I'll be updating the Buildbot masters to pick this up today or tomorrow (depends when they are not so busy).

In the meantime I can get new configs ready. John, where do you want success mail sent to?
For now, "build@mozilla.org". 

Once we shake out any last minute hiccups, and have had time to forewarn everyone, we can change to "release-drivers@mozilla.org", but lets not do that just yet.
Attachment #299257 - Flags: review?(rhelmer) → review+
Comment on attachment 299257 [details] [diff] [review]
[checked in] add new MailNotifier to config

Checking in staging/master.cfg;
/cvsroot/mozilla/tools/buildbot-configs/automation/staging/master.cfg,v  <--  master.cfg
new revision: 1.23; previous revision: 1.22
done
Checking in staging-1.9/master.cfg;
/cvsroot/mozilla/tools/buildbot-configs/automation/staging-1.9/master.cfg,v  <--  master.cfg
new revision: 1.21; previous revision: 1.20
done
Attachment #299257 - Attachment description: add new MailNotifier to config → [checked in] add new MailNotifier to config
Attached file cvs tag log
I've re-tagged mozilla/tools/buildbot on BUILDBOT_0_7_5_BRANCH with BUILDBOT_MOZILLA_PROUDUCTION. This is the same code we're running now + the MailNotifier patch. The tag will need to be moved if/when we land more code on this branch.
The production masters will need to have Buildbot updated for this patch to work. Here's what I did:

ssh root@host

export PYTHONHOME="/tools/python"
export PATH="/tools/python/bin:$PATH"
export PYTHONPATH=.:$PYTHONPATH
cd /tools
cvs -d:ext:stgbld@cvs.mozilla.org:/cvsroot co -d buildbot-production -r BUILDBOT_MOZILLA_PRODUCTION mozilla/tools/buildbot
python setup.py install --prefix=/tools/buildbot-production
rm -f buildbot-trunk && ln -s buildbot-production buildbot
Attachment #299778 - Flags: review?(rhelmer) → review+
Comment on attachment 299778 [details] [diff] [review]
[checked in] add the new MailNotifier to the production automation configs

Checking in master.cfg;
/cvsroot/mozilla/tools/buildbot-configs/automation/production-1.9/master.cfg,v  <--  master.cfg
new revision: 1.8; previous revision: 1.7
done
Checking in production/master.cfg;
/cvsroot/mozilla/tools/buildbot-configs/automation/production/master.cfg,v  <--  master.cfg
new revision: 1.15; previous revision: 1.14
done
Attachment #299778 - Attachment description: add the new MailNotifier to the production automation configs → [checked in] add the new MailNotifier to the production automation configs
Status: ASSIGNED → RESOLVED
Closed: 12 years ago
Resolution: --- → FIXED
Sorry for the bugspam, I got too excited when closing bugs -- this still needs work.
Status: RESOLVED → REOPENED
Resolution: FIXED → ---
We've talked about what to do here on IRC. My preferred method is to enhance the MailNotifier to take a message generator function. It's signature would look something like this (with perhaps another argument):
def generateMessage(IBuildStatus)

I'm busy right now, but intend to do this after getting some other things out of the way.

There's an upstream bug filed about this:
http://buildbot.net/trac/ticket/175
Priority: P2 → P3
Summary: Automation should send nicer emails → use automation, not humans, to send notification emails to other groups
Status: REOPENED → ASSIGNED
Priority: P3 → P2
Priority: P2 → P3
Buildbot 0.7.7 is taking priority here...back to the queue.
Assignee: bhearsum → nobody
Status: ASSIGNED → NEW
QA Contact: build → release
John, we don't have the 'passing' mode patch landed on 0.7.7. Given that this is not doing what we want I was thinking we may as well just disable the 'passing' mode MailNotifier. What do you think?
Assignee: nobody → joduinn
urgh - while I still think this is something we should do, I've not had time to look at this in months. Taking my own medicine, and pushing to Future until I (or anyone else) gets time for this.

fyi: Recent work by calee and in buildbot 0.7.10 should make this easier to do now.
Assignee: joduinn → nobody
Component: Release Engineering → Release Engineering: Future
Duplicate of this bug: 394507
Status: NEW → ASSIGNED
Component: Release Engineering: Future → Release Engineering
Assignee: nobody → armenzg
Priority: P3 → P2
Priority: P2 → P3
(In reply to comment #28)
> urgh - while I still think this is something we should do, I've not had time to
> look at this in months. Taking my own medicine, and pushing to Future until I
> (or anyone else) gets time for this.
> 
> fyi: Recent work by calee and in buildbot 0.7.10 should make this easier to do
> now.

Any update on this?
It is half of the quarter and I have not even touched this goal and the mobile-L10n infra still needs few more weeks. Let me know what to do with this.
I haven't had time to work on this - putting back in the pool.
I talked with John about it.
Assignee: armenzg → nobody
Status: ASSIGNED → NEW
Component: Release Engineering → Release Engineering: Future
Priority: P3 → --
Assignee: nobody → bhearsum
Whiteboard: [automation]
So, after re-reading this bug it seems like all of the technical pieces are in place to make this possible. So we need to decide on exactly what mail needs to be sent, and work out the text of them. Traditionally, we send the following mail as part of the automated release process:
* "Tagging started"
* "X builds available" (where X is Linux, Mac, and Signed Win32)
* "Updates available on betatest"

I'm also going to have a look over old release notes / logs to see if there's any more error catching we can do to avoid sending r-d mail when something is wrong.
Mass move of bugs from Release Engineering:Future -> Release Engineering. See http://coop.deadsquid.com/2010/02/kiss-the-future-goodbye/ for more details.
Component: Release Engineering: Future → Release Engineering
Priority: -- → P3
It seems doubtful that I'll be touching this anytime soon
Assignee: bhearsum → nobody
Priority: P3 → P5
Assignee: nobody → rail
Could we set up an ftp poller script that would send out email to release-drivers when each OS shows up in the candidates dir for the first time?
(In reply to comment #36)
> Could we set up an ftp poller script that would send out email to
> release-drivers when each OS shows up in the candidates dir for the first time?

That would need to poll when the *last* locale shows up, which can be tricky. Instead maybe have buildbot master send the email if the particular step is completed and green?
(In reply to comment #33)
> So, after re-reading this bug it seems like all of the technical pieces are in
> place to make this possible. So we need to decide on exactly what mail needs to
> be sent, and work out the text of them. Traditionally, we send the following
> mail as part of the automated release process:
> * "Tagging started"
> * "X builds available" (where X is Linux, Mac, and Signed Win32)
> * "Updates available on betatest"
> 
> I'm also going to have a look over old release notes / logs to see if there's
> any more error catching we can do to avoid sending r-d mail when something is
> wrong.

After some irc discussions, here a strawman proposal of what emails to automate
sending. Note: sending email to release-drivers unless noted otherwise.

* Tagging started
* Tagging ended - ok to reopen the tree (maybe only to release@m.c?)
* Builds available for linux32
* Builds available for linux64
* Builds available for osx10.5
* Builds available for osx10.6
* Builds available for win32 (unsigned)
* Builds available for win32 (signed)
* l10n-verify-for-all-OS (only to release@m.c; do separate email per OS if
easier?)
* updates available on betatest
* Update-verify-for-all-OS (only to release@m.c; do separate email per OS if
easier?)


There are other emails we send which I feel could be automated, but maybe these
should be tracked in bug#394498? 
* mirror push started
* mirrors absorbed enough for QA to start testing
* mirrors absorbed enough for "normal" release


Anything I missed? Any comments or concerns?
what about the antivirus vendors?
Attached patch [tested]release_notifier_patch (obsolete) — Splinter Review
Based off the work in upgrading desktop release masters to 0.8.0 in bug 553300, I've added a simple set of Mail/Change Notifiers that follow the outline in John's comment. The subject/body text of the messages is based off of what I've seen in my inbox during releases, and may need to be modified, and we may want to send somewhere other than build@m.o.
Attachment #480767 - Flags: feedback?(aki)
Comment on attachment 480767 [details] [diff] [review]
[tested]release_notifier_patch

>+            msgdict['body'] = """
>+Tagging complete for %s build%s (OK to re-open tree)
>+
>+-buildbot
>+""" % (baseTag, buildNumber) if not results else """Tagging failed for %s build%s""" % (baseTag, buildNumber)

Nit: this is a bit messy.  Maybe due to frustrations w/ perl and its multi-faceted ways of creating statements, I prefer having nested if's rather than complex statements, e.g.

  if results:
    ...
  else:
    ...

The msgdict['body'] section in general with its multi-lines isn't ideal, but I'm not sure creating lots of little template files is the best route either atm.
So I'm noting it here, but we probably don't have to change it at this point.

>+            extraRecipients=['build@mozilla.com',],

This should probably be release@m.c, and should be configurable in release-*.py at some point, but this is probably fine until we decide to send to an external audience.

Then again, if multiple people are running staging builds simultaneously, having easily configurable email addresses might be important.

I'd like to see this in a staging release run at some point; whoever's next up to do one.

All in all, thanks for taking this on!
Attachment #480767 - Flags: feedback?(aki) → feedback+
Attached patch [tested]release_notifier_patch (obsolete) — Splinter Review
Implementing configurable recipients was actually fairly trivial, so I've gone ahead and done that. Cleaning up the releaseMessage functions is still a bit hairy. In spite of aki's suggestion, that code is still pretty unreadable. Perhaps the body text should also be moved into releaseConfigs? 

The way I tested this was to run a staging release that didn't actually do anything (i.e. Dummy builders for every step, no signing, force builds for post-signing steps). Riding along on a proper staging release would be good, but may not be necessary.
Assignee: rail → salbiz
Attachment #480767 - Attachment is obsolete: true
Attachment #480939 - Flags: review?(rail)
Comment on attachment 480939 [details] [diff] [review]
[tested]release_notifier_patch

Great job! 

But :)
* Looks like the patch was produced against not yet published version of buildbotcustom (0.8 migration, I think). Please add a dependency for the current bug.
* Instead of using if/else for different types of builders I'd prefer to use a template ("subject\n\nbody<EOF>", where subject and body can contain %(version)s style substitution strings) driven approach with a fallback template if no specific template found. In this case we can easily adjust the message subject/body without reconfiguring the master.
* platform name should be provided for *_build builder notifications.
* instead of using releaseConfig['baseTag'].title() I'd prefer to use "%s %s" % (releaseConfig['productName'].title(), releaseConfig['version'])
* No notification after ftppoller detects signed builds
* build@mozilla.org is not a valid address, please use release@
* Probably we can filter builders going to release@ by category='release', so we'll get notifications for all release related builders, even l10n repacks. In this case notify_builders will be used for mode='passing' notifications.
Attachment #480939 - Flags: review?(rail) → review-
Depends on: 553300
Attached patch [tested]release_notifier_patch (obsolete) — Splinter Review
(In reply to comment #43)
> Comment on attachment 480939 [details] [diff] [review]
> [tested]release_notifier_patch
> 
> Great job! 
> 
> But :)
> * Looks like the patch was produced against not yet published version of
> buildbotcustom (0.8 migration, I think). Please add a dependency for the
> current bug.
Yep, you're right. Done.

> * Instead of using if/else for different types of builders I'd prefer to use a
> template ("subject\n\nbody<EOF>", where subject and body can contain
> %(version)s style substitution strings) driven approach with a fallback
> template if no specific template found. In this case we can easily adjust the
> message subject/body without reconfiguring the master.
I agree, but I'm not too sure where to put these templates. In this patch, I've just dumped them into a nested hash inside 'generateReleaseBranchObjects', but I don't think that's a good place for them

> * platform name should be provided for *_build builder notifications.
Done, currently, I differentiate the xulrunner builds to make the emails a bit clearer. We may want to change this behaviour (I don't think I've ever seen anyone send a notification to r-d about xulrunner builds)

> * instead of using releaseConfig['baseTag'].title() I'd prefer to use "%s %s" %
> (releaseConfig['productName'].title(), releaseConfig['version'])
Done, this is much nicer.

> * No notification after ftppoller detects signed builds
Added a change notifier to the post_signing branch to fire off an email for this case

> * build@mozilla.org is not a valid address, please use release@
Done

> * Probably we can filter builders going to release@ by category='release', so
> we'll get notifications for all release related builders, even l10n repacks. In
> this case notify_builders will be used for mode='passing' notifications.
Done.

I think I've ironed out most of the problems with this patch, I'm just a bit unsure about where to put the message templates and the actual text the messages should contain.
Attachment #480939 - Attachment is obsolete: true
Attachment #481360 - Flags: feedback?(rail)
Comment on attachment 481360 [details] [diff] [review]
[tested]release_notifier_patch

> I agree, but I'm not too sure where to put these templates.

Add "templates" directory and put the templates there. :)

Something like this:

templates
├── build_failed.txt
├── build_success.txt
├── signing_success.txt
├── tag_change.txt
├── tag_failed.txt
├── tag_success.txt
├── win32_build_failed.txt
└── win32_build_success.txt

Template search logic may vary.

For platform independent steps:
%s(step)s_%s(job_status)s.txt -> %s(job_status)s.txt -> default.txt

For platform dependent steps:
%s(platform)s_%s(step)s_%s(job_status)s.txt -> %s(step)s_%s(job_status)s.txt -> %s(job_status)s.txt -> default.txt

builder-masters and universal-masters should be adjusted in master-setup.py to symlink this directory.

>+        if name.endswith("-tag"):
>+            step = "tag"
>+        elif name.endswith("_build"):
>+            step = "build"
>+        else:
>+            step = "other"
>+
>+        (subject, body) = templates[step][job_status]

I'd prefer more flexible way to extend templates.

If you want to add another step you need to edit the code in 2 places (template itself + if/else clause above).

You can use try/catch to get subject/body or fall back to the default template


>+            builders=[b['name'] for b in builders if b['category'] == 'release'],

I meant categories=['release'] instead of builders (see "MailNotifier arguments" in buildbot docs).

> I think I've ironed out most of the problems with this patch, I'm just a bit
> unsure about where to put the message templates and the actual text the
> messages should contain.

You can just copy the actual messages sent to release-drivers@ mailing list.

Overall looks fine. feedback+

Thanks a lot for grabbing this bug from me, feel free to bother me if you have questions! :)
Attachment #481360 - Flags: feedback?(rail) → feedback+
Attached patch [tested]release_notifier_patch (obsolete) — Splinter Review
Attachment #481360 - Attachment is obsolete: true
Attachment #481650 - Flags: review?(rail)
These two patches implement the message bodies with the template system you described. Thanks for explaining it for me. :)
The default release_config still does not include r-d@, but since it is configurable I am not too worried.
Attachment #481659 - Flags: review?(rail)
here is an interdiff between the last patch and this one.
Attachment #481659 - Flags: review?(rail) → review+
Comment on attachment 481650 [details] [diff] [review]
[tested]release_notifier_patch

Great progress!

>+        try:
>+            if platform:
>+                try:
>+                    template = open("%s/%s_%s_%s" % (releaseConfig['releaseTemplates'], platform, stage, job_status), "r", True)
>+                except IOError:
>+                    template = open("%s/%s_%s" % (releaseConfig['releaseTemplates'], stage, job_status), "r", True)
>+            else:
>+                template = open("%s/%s_%s" % (releaseConfig['releaseTemplates'], stage, job_status), "r", True)
>+        except IOError:
>+            template = open("%s/default_%s" % (releaseConfig['releaseTemplates'], job_status), "r", True)

r+ if you change this hunk to something similar to the following snippet, which IMHO is more readable:

possible_template_names = ("%s/%s_%s_%s" % (releaseConfig['releaseTemplates'], platform, stage, job_status),
                           "%s/%s_%s" % (releaseConfig['releaseTemplates'], stage, job_status),
                           "%s/default_%s" % (releaseConfig['releaseTemplates'], job_status))
template = None

for t in possible_template_names: 
    if check_if_template_exists():
        template = open(t, "r", True)

if template:
   ...


I also will be glad to take a look at the working copy of this in staging.
Attachment #481650 - Flags: review?(rail) → review+
Done, did a quick staging run with dummy builders to make sure it works. So far the behavior in case of not fallback template is still the same as before (raise an IOError). I think the twistd logger should catch these if they occur.
Attachment #481650 - Attachment is obsolete: true
Attachment #481661 - Attachment is obsolete: true
Attachment #482754 - Flags: review?(rail)
Comment on attachment 482754 [details] [diff] [review]
[tested] tweak template selection

Great! /me wants to see this in action :)
Attachment #482754 - Flags: review?(rail) → review+
Whilst setting up staging I noticed that because of some of the changes I made, the functionality was a little broken. Firstly, if the job specific templates weren't there it would fall back to the default template without considering the platform, and the update_verify builders were not being counted properly. This patch fixes those two problems. Interdiff is only two lines
Attachment #482754 - Attachment is obsolete: true
Attachment #483172 - Flags: review?(rail)
Attached patch interdiff (obsolete) — Splinter Review
interdiff for that last change, since it's only two lines that change
Attachment #483172 - Flags: review?(rail) → review+
Attachment #483172 - Attachment is obsolete: true
Attachment #483174 - Attachment is obsolete: true
Attachment #483199 - Flags: review?(rail)
Attachment #483199 - Flags: checked-in?
Attachment #483199 - Flags: review?(rail) → review+
since 0.8.0 landed, this should be ok to land as well.
Attachment #481659 - Attachment is obsolete: true
Comment on attachment 483199 [details] [diff] [review]
use xulrunnerPlatforms to populate platforms list for XR

changeset:   1176:552b02de6346
Attachment #483199 - Flags: checked-in? → checked-in+
Comment on attachment 486918 [details] [diff] [review]
fixing bitrot, adding releaseConfig items to staging configs as well.

changeset:   3296:218345ab5634
Attachment #486918 - Flags: checked-in+
We'll let this bake for now, sending mail to release@ only for the next set of releases. They'll probably be some tweaking to the messages we want to do.
I hit an Exception in staging that seems related to this:
	    w.buildFinished(name, s, results)
	  File "/tools/buildbot-0.8.0/lib/python2.6/site-packages/buildbot-0.8.1-py2.6.egg/buildbot/status/mail.py", line 356, in buildFinished
	    return self.buildMessage(name, build, results)
	  File "/tools/buildbot-0.8.0/lib/python2.6/site-packages/buildbot-0.8.1-py2.6.egg/buildbot/status/mail.py", line 466, in buildMessage
	    msgdict = self.messageFormatter(self.mode, name, build, results, self.master_status)
	  File "/builds/buildbot/bhearsum/buildbotcustom/process/release.py", line 72, in createReleaseMessage
	    if template:
	exceptions.UnboundLocalError: local variable 'template' referenced before assignment
	
It looks like the code that tries to find the correct template doesn't fall back gracefully if it can't find one.
Comment on attachment 486918 [details] [diff] [review]
fixing bitrot, adding releaseConfig items to staging configs as well.

Turns out I forgot to add the templates until just now :(. They landed in changeset:   3331:b8330af16ca3
Depends on: 613226
Once bug 614943 gets fixed, we can flip the switch to direct mail to r-d@ by default.
Attachment #500340 - Flags: review?(bhearsum)
Comment on attachment 500340 [details] [diff] [review]
flip the switch to r-d

Landed this on default, planning to reconfig later.
Attachment #500340 - Flags: review?(bhearsum) → review+
Comment on attachment 500340 [details] [diff] [review]
flip the switch to r-d

Merged to production and masters have been reconfiged.
Attachment #500340 - Flags: checked-in+
Great work here Syed, we're all done!
Status: NEW → RESOLVED
Closed: 12 years ago9 years ago
Resolution: --- → FIXED
Blocks: 622945
Product: mozilla.org → Release Engineering
You need to log in before you can comment on or make changes to this bug.