763929 - tracking bug for initial implementation + deployment of release kickoff and release runner

Status: RESOLVED FIXED

(Release Engineering :: Release Automation: Other, defect, P3)

Description

[[e-mail] (:bhearsum)]

There's a bunch of things we need to do to start a release. They're very standarized these days, and rely on a few different inputs. We should script it. Specifically, the tool needs to:
* Download a reviewed release config patch, double land it in a local repository
* Download l10n-changesets from the dashboard
* Tag buildbot-configs, buildbotcustom, and tools with the release tags
* Set clobbers
* Set reserved_slaves
* Update and reconfig the relevant master
* Run release sanity in dry run mode
* Run release sanity to start the automation

All the parts that involve interacting with the master could probably be done by enhancing our existing Fabric script. The rest is very simple scripting.

Comment 1

[Armen [:armenzg]]

Attachment: tagging repos script

This is a tagging repos script I've always used (a script I should have improved so I would not feel embarrassed when attaching it).


BTW [phone] to this bug

Comment 2

[[e-mail] (:bhearsum)]

We should make this script e-mail metrics, too.

Comment 3

[[e-mail] (:bhearsum)]

Hope we can get to this soon, it's not higher priority than some other things though.

Comment 4

[[e-mail] (:bhearsum)]

Rail and I got a big start on this last week. There's a webapp over in https://github.com/bhearsum/release-kickoff, and a bunch of modifications to existing scripts in https://github.com/bhearsum/tools/compare/master...kickoff

There's still more to be done, details over in this etherpad: https://etherpad.mozilla.org/ReleaseKickOff

Comment 5

[[e-mail] (:bhearsum)]

Attachment: add release config templates

Comment 6

[[e-mail] (:bhearsum)]

still need templates for release/esr configs

Comment 7

[[e-mail] (:bhearsum)]

Attachment: release kickoff webapp

It occurs to me that having some review before we're almost ready to land would be smart. This patch is the release kickoff app. Not sure who should review it though.

Comment 8

[[e-mail] (:bhearsum)]

Attachment: release runner + tool modifications

Most of this was done by Rail but also includes the release kickoff api that I wrote. Catlee knows most of these tools pretty well, so I choose him!

Comment 9

[Curtis Koenig [:curtisk-use curtis.koenig+bzATgmail.com]]]

security review scheduled: review scheduled:https://mail.mozilla.com/home/[e-mail]/Security%20Review.html?view=month&action=view&invId=110484-110483&pstat=AC&exInvId=110484-239613&useInstance=1&instStartTime=1353002400000&instDuration=3600000

Comment 10

[[e-mail] (:bhearsum)]

Attachment: create templates for esr10

Comment 11

[[e-mail] (:bhearsum)]

Attachment: release channel templates

Comment 12

[Aki Sasaki (not active)]

In production.

Comment 13

[[e-mail] (:bhearsum)]

Comment on attachment 677553 [details] [diff] [review]
release kickoff webapp

We're going to do reviews in person next week.

Comment 14

[[e-mail] (:bhearsum)]

Attachment: release kickoff webapp

Chris, here's a diff of release kickoff (vendor library excluded) with your initial review comments addressed. For posterity, here's what you said in e-mail:
- Use @classmethod instead of @staticmethod in model
- SubmitRelease.post should have a comment that says REMOTE_USER is being checked for elsewhere
  same with Releases.post

I also added a bunch of comments and fixed some unused imports.

Comment 15

[[e-mail] (:bhearsum)]

Attachment: release runner + friends

Addresses these comments that you made, except where noted below:
- Remove lib/python/vendor from site.addsitedir in manage_masters
- Move update() and symlink calls in release-sanity up inside previous else block
- Comment why need list(reversed(options.releaseConfigFiles))
- Move unlink() calls into a check against which configs dir we're using. Why do we need to unlink "localconfig.py"? For clean commits? Why not use hg commit file...
- remove duplicate sendchange_master = config.get('...') and fix error handling if it's not set
- release runner needs comments; e.g. what while loop is doing
- refactor code like
    for release in rr.new_releases:
        log.releaseName = ...
-- I refactored the main part of release runner's code, but the specific thing about log.releaseName isn't relevant, because that work is being postphoned.

- refactor tagging calls
-- Not sure what this means
- look at having log handlers per release instead of one common log handler
-- This got postphoned, too.
- look at buildbot sendchange --help to figure out if we should use --who or --username
- pass Popen object to run_cmd_poll callback

Comment 16

[[e-mail] (:bhearsum)]

Attachment: release runner + friends

Found a bug in the buildbotcustom/tools tagging section today (it tries to tag the repositories multiple times, resulting in failures during a build1 with more than one release).

Comment 17

[Chris AtLee [:catlee]]

Comment on attachment 696723 [details] [diff] [review]
release runner + friends

Review of attachment 696723 [details] [diff] [review]:
-----------------------------------------------------------------

::: buildbot-helpers/release_sanity.py
@@ +425,5 @@
> +
> +    # https://bugzilla.mozilla.org/show_bug.cgi?id=678103#c5
> +    # This goes through the list of config files in reverse order, which is a
> +    # hacky way of making sure that the config file that's listed first is the
> +    # one that's loaded in releaseConfig for the sendchange.

Why does this matter? The only thing that the sendchange uses from the releaseConfig is the tag name, but the tag for any product should be equivalent, right?

::: buildfarm/release/release-runner.sh
@@ +1,4 @@
> +#!/bin/bash
> +
> +# Sleep 3 days in case of failure
> +SLEEP_TIME=259200

3 days? really?

Comment 18

[[e-mail] (:bhearsum)]

(In reply to Chris AtLee [:catlee] from comment #17)
> Comment on attachment 696723 [details] [diff] [review]
> release runner + friends
> 
> Review of attachment 696723 [details] [diff] [review]:
> -----------------------------------------------------------------
> 
> ::: buildbot-helpers/release_sanity.py
> @@ +425,5 @@
> > +
> > +    # https://bugzilla.mozilla.org/show_bug.cgi?id=678103#c5
> > +    # This goes through the list of config files in reverse order, which is a
> > +    # hacky way of making sure that the config file that's listed first is the
> > +    # one that's loaded in releaseConfig for the sendchange.
> 
> Why does this matter? The only thing that the sendchange uses from the
> releaseConfig is the tag name, but the tag for any product should be
> equivalent, right?
> 
> ::: buildfarm/release/release-runner.sh
> @@ +1,4 @@
> > +#!/bin/bash
> > +
> > +# Sleep 3 days in case of failure
> > +SLEEP_TIME=259200
> 
> 3 days? really?

302 rail on these

Comment 19

[Rail Aliiev [:rail]]

(In reply to Chris AtLee [:catlee] from comment #17)
> Comment on attachment 696723 [details] [diff] [review]
> release runner + friends
> 
> Review of attachment 696723 [details] [diff] [review]:
> -----------------------------------------------------------------
> 
> ::: buildbot-helpers/release_sanity.py
> @@ +425,5 @@
> > +
> > +    # https://bugzilla.mozilla.org/show_bug.cgi?id=678103#c5
> > +    # This goes through the list of config files in reverse order, which is a
> > +    # hacky way of making sure that the config file that's listed first is the
> > +    # one that's loaded in releaseConfig for the sendchange.
> 
> Why does this matter? The only thing that the sendchange uses from the
> releaseConfig is the tag name, but the tag for any product should be
> equivalent, right?

I've just copied the explanation from the corresponding bug. However, it looks like that part can be simplified.
 
> ::: buildfarm/release/release-runner.sh
> @@ +1,4 @@
> > +#!/bin/bash
> > +
> > +# Sleep 3 days in case of failure
> > +SLEEP_TIME=259200
> 
> 3 days? really?

per https://bugzilla.mozilla.org/show_bug.cgi?id=810395#c0
  * sleep 3 days (in case of weekend)

It gives us enough time to fix the problem and restart the service. Otherwise the service would be started again by supervisord and probably fail again.

Comment 20

[[e-mail] (:bhearsum)]

Now that we've farmed off all the remaining tasks to other bugs, updating the summary to reflect that this is a tracking bug.

Comment 21

[[e-mail] (:bhearsum)]

Comment on attachment 696076 [details] [diff] [review]
release kickoff webapp

This has been landed for awhile.

Comment 22

[[e-mail] (:bhearsum)]

Attachment: fix failing tests

We had some test failures in release runner upon landing:
======================================================================
ERROR: testSuccess3secsWith2secsPoll (mozilla_buildtools.test.test_util_commands.TestRunCmdiPeriodicPoll)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/builds/buildbot/preproduction/slave/test-masters/tools/lib/python/mozilla_buildtools/test/test_util_commands.py", line 61, in testSuccess3secsWith2secsPoll
    warning_interval=2),
  File "/builds/buildbot/preproduction/slave/test-masters/tools/lib/python/util/commands.py", line 111, in run_cmd_periodic_poll
    elapsed))
TypeError: %d format: a number is required, not NoneType
-------------------- >> begin captured logging << --------------------
util.commands: INFO: command: START
util.commands: INFO: command: bash -c "sleep 3 && true"
util.commands: INFO: command: cwd: /builds/buildbot/preproduction/slave/test-masters/tools/lib/python
util.commands: INFO: command: output:
--------------------- >> end captured logging << ---------------------

======================================================================
FAIL: testMakeHGUrl (mozilla_buildtools.test.test_util_hg.TestHg)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/builds/buildbot/preproduction/slave/test-masters/tools/lib/python/mozilla_buildtools/test/test_util_hg.py", line 416, in testMakeHGUrl
    self.assertEquals(file_url, expected_url)
AssertionError: 'https://hg.mozilla.org/build/tools/raw-file/FIREFOX_3_6_12_RELEASE/lib/python/util/hg.py' != 'http://hg.mozilla.org/build/tools/raw-file/FIREFOX_3_6_12_RELEASE/lib/python/util/hg.py'
-------------------- >> begin captured logging << --------------------
util.commands: INFO: command: START
util.commands: INFO: command: /builds/buildbot/preproduction/slave/test-masters/tools/lib/python/mozilla_buildtools/test/init_hgrepo.sh /tmp/tmpw7IHXC/repo
util.commands: INFO: command: cwd: /builds/buildbot/preproduction/slave/test-masters/tools/lib/python
util.commands: INFO: command: output:
util.commands: INFO: command: END (0.65s elapsed)

util.commands: INFO: command: START
util.commands: INFO: command: hg log -R /tmp/tmpw7IHXC/repo --template {node|short}

util.commands: INFO: command: cwd: /builds/buildbot/preproduction/slave/test-masters/tools/lib/python
util.commands: INFO: command: output:
util.commands: INFO: 8a9b4d8790e3
c877587e5685
c9d4cab6dd85
3c3f19b1c095

util.commands: INFO: command: END (0.09 elapsed)

--------------------- >> end captured logging << ---------------------

======================================================================
FAIL: testMakeHGUrlNoFilename (mozilla_buildtools.test.test_util_hg.TestHg)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/builds/buildbot/preproduction/slave/test-masters/tools/lib/python/mozilla_buildtools/test/test_util_hg.py", line 425, in testMakeHGUrlNoFilename
    self.assertEquals(file_url, expected_url)
AssertionError: 'https://hg.mozilla.org/build/tools/rev/default' != 'http://hg.mozilla.org/build/tools/rev/default'
-------------------- >> begin captured logging << --------------------
util.commands: INFO: command: START
util.commands: INFO: command: /builds/buildbot/preproduction/slave/test-masters/tools/lib/python/mozilla_buildtools/test/init_hgrepo.sh /tmp/tmp1caXNg/repo
util.commands: INFO: command: cwd: /builds/buildbot/preproduction/slave/test-masters/tools/lib/python
util.commands: INFO: command: output:
util.commands: INFO: command: END (0.94s elapsed)

util.commands: INFO: command: START
util.commands: INFO: command: hg log -R /tmp/tmp1caXNg/repo --template {node|short}

util.commands: INFO: command: cwd: /builds/buildbot/preproduction/slave/test-masters/tools/lib/python
util.commands: INFO: command: output:
util.commands: INFO: 61ecdbf8ea00
d0ffc2c2c3f4
200b6339415e
c57d88da3e99

util.commands: INFO: command: END (0.11 elapsed)

--------------------- >> end captured logging << ---------------------

======================================================================
FAIL: testMakeHGUrlNoRevisionNoFilename (mozilla_buildtools.test.test_util_hg.TestHg)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/builds/buildbot/preproduction/slave/test-masters/tools/lib/python/mozilla_buildtools/test/test_util_hg.py", line 433, in testMakeHGUrlNoRevisionNoFilename
    self.assertEquals(repo_url, expected_url)
AssertionError: 'https://hg.mozilla.org/build/tools' != 'http://hg.mozilla.org/build/tools'
-------------------- >> begin captured logging << --------------------
util.commands: INFO: command: START
util.commands: INFO: command: /builds/buildbot/preproduction/slave/test-masters/tools/lib/python/mozilla_buildtools/test/init_hgrepo.sh /tmp/tmpUpxA4h/repo
util.commands: INFO: command: cwd: /builds/buildbot/preproduction/slave/test-masters/tools/lib/python
util.commands: INFO: command: output:
util.commands: INFO: command: END (0.98s elapsed)

util.commands: INFO: command: START
util.commands: INFO: command: hg log -R /tmp/tmpUpxA4h/repo --template {node|short}

util.commands: INFO: command: cwd: /builds/buildbot/preproduction/slave/test-masters/tools/lib/python
util.commands: INFO: command: output:
util.commands: INFO: c2d24c97c173
ab42e8e02cf7
3384239335c1
a7a2846df2f1

util.commands: INFO: command: END (0.11 elapsed)

--------------------- >> end captured logging << ---------------------

----------------------------------------------------------------------
Ran 312 tests in 177.151s

FAILED (errors=1, failures=3)

One of them seems to repro'ing the problem I hit in bug 828023.

Comment 23

[[e-mail] (:bhearsum)]

The initial version of the web app and release runner are both deployed to production now. Follow up issues and improvements will be tracked in individual bugs marked with "[shipit]" in the whiteboard. Huge thanks to everyone who helped out with this!