836289 - puppetize release runner deployment

Status: RESOLVED FIXED

(Release Engineering :: Release Automation, defect)

Description

[[e-mail] (:bhearsum)]

bug 810393 did deployment of release runner by hand, because the master it's running on doesn't have a properly functioning puppet (bug 777742). Once that's fixed we need to puppetize release runner.

Comment 1

[Rail Aliiev [:rail]]

P3 for now, not working on it actively, blocked by another bug.

Comment 2

[Rail Aliiev [:rail]]

back to the pool

Comment 3

[[e-mail] (:bhearsum)]

Rail, is your manual deployment documented somewhere? If not, can you provide details here?

Comment 4

[Rail Aliiev [:rail]]

That's Bug 810393.
* /etc/supervisor.conf can be grabbed from bm36
* no need to checkout from git anymore

Comment 5

[Rail Aliiev [:rail]]

Attachment: puppetize release-runner

* not tested at all

Comment 6

[Dustin J. Mitchell [:dustin] (he/him)]

Comment on attachment 764192 [details] [diff] [review]
puppetize release-runner

Review of attachment 764192 [details] [diff] [review]:
-----------------------------------------------------------------

It doesn't look like you actually declare the releaserunner class anywhere.

I assume this will be a new toplevel class that just gets included in the node def along with the buildmaster toplevel?

::: manifests/extlookup/secrets-template.csv
@@ +51,5 @@
>  win_tests_password,""
> +releaserunner_api_username,""
> +releaserunner_api_password,""
> +releaserunner_ssh_username,""
> +releaserunner_ssh_password,""

password? itym private key?

::: modules/mercurial/manifests/repos.pp
@@ +1,4 @@
> +# This Source Code Form is subject to the terms of the Mozilla Public
> +# License, v. 2.0. If a copy of the MPL was not distributed with this
> +# file, You can obtain one at http://mozilla.org/MPL/2.0/.
> +define mercurial::repos($hg_repo, $dst_dir, $branch="default", $user='') {

The name should probably be singular (mercurial::repo)

@@ +2,5 @@
> +# License, v. 2.0. If a copy of the MPL was not distributed with this
> +# file, You can obtain one at http://mozilla.org/MPL/2.0/.
> +define mercurial::repos($hg_repo, $dst_dir, $branch="default", $user='') {
> +    include packages::mozilla::py27_mercurial
> +    include users::builder

It'd be better to put this inside a conditional, so that we can use mercurial::repos on a host without a builder user (e.g. an imaging server).

::: modules/releaserunner/manifests/services.pp
@@ +16,5 @@
> +logfile_maxbytes=50MB
> +logfile_backups=10
> +; once we upgrade tto 3.0b2: stopasgroup=true
> +; once we upgrade tto 3.0b2: killasgroup=true
> +"

This string might do better as a template -
  $extra_config = template("${module_name}/extra_config.erb")

Comment 7

[Rail Aliiev [:rail]]

(In reply to Dustin J. Mitchell [:dustin] from comment #6)
> It doesn't look like you actually declare the releaserunner class anywhere.
> 
> I assume this will be a new toplevel class that just gets included in the
> node def along with the buildmaster toplevel?

Yes. One of the masters should just work.
 
> > +releaserunner_ssh_password,""
> 
> password? itym private key?

Yup. A silly calque...
  
> The name should probably be singular (mercurial::repo)

+1
 
> It'd be better to put this inside a conditional, so that we can use
> mercurial::repos on a host without a builder user (e.g. an imaging server).

Agree.

> This string might do better as a template -
>   $extra_config = template("${module_name}/extra_config.erb")

Good idea.

I'll update the patch after I test it (probably next week after releases).

Thanks!

Comment 8

[Rail Aliiev [:rail]]

Attachment: tools

pass virtualenv directory and logfile to release-runner.sh and fallback to bm36 values by default.

Comment 9

[Rail Aliiev [:rail]]

Attachment: puppetize release-runner

* address comments
* full list of pip packages
* add gcc/make/mysql-devel so fabric/buildbot are installed properly
* it works! :)

Comment 10

[Rail Aliiev [:rail]]

Comment on attachment 768449 [details] [diff] [review]
tools

https://hg.mozilla.org/build/tools/rev/db090f1d4c67

deployed on bm36 (and properly restarted)

Comment 11

[Dustin J. Mitchell [:dustin] (he/him)]

Comment on attachment 768459 [details] [diff] [review]
puppetize release-runner

Review of attachment 768459 [details] [diff] [review]:
-----------------------------------------------------------------

Just a few minor changes to the config, and this will be good to go.

::: modules/config/manifests/base.pp
@@ +46,5 @@
> +    $releaserunner_notify_from = "Release Eng <[e-mail]>"
> +    $releaserunner_notify_to = "Release Eng <[e-mail]>"
> +    $releaserunner_smtp_server = "localhost"
> +    $releaserunner_hg_username = "ffxbld"
> +    $releaserunner_hg_ssh_key = "/home/cltbld/.ssh/ffxbld_dsa"

It doesn't make a lot of sense to use config to point to a local file which isn't managed by puppet.  Can you make this (and the $releaserunner_ssh_key below) settings for now, with plans to actually manage those keys directly at a later time?

@@ +56,5 @@
> +    $releaserunner_tools_branch = "default"
> +    $releaserunner_production_masters = "http://hg.mozilla.org/build/tools/raw-file/default/buildfarm/maintenance/production-masters.json"
> +    $releaserunner_sendchange_master = "buildbot-master36.srv.releng.scl3.mozilla.com:9301"
> +    $releaserunner_ssh_username = "cltbld"
> +    $releaserunner_ssh_key = "/home/cltbld/.ssh/release-runner"

Many of the defaults here are moco-specific (basically, everything but the repos/branches).  Can you move those values to moco-config.pp?

::: modules/mercurial/manifests/repo.pp
@@ +5,5 @@
> +    include packages::mozilla::py27_mercurial
> +
> +    if $user == '' {
> +        include users::builder
> +        $hg_user = $users::builder::username

This reference would technically need an `include users::builder`, but then you couldn't use a mercurial repo without the builder, which would be problematic on e.g., signing servers.  Better to just always pass this argument, I think.

::: modules/releaserunner/manifests/init.pp
@@ +36,5 @@
> +            ];
> +    }
> +
> +    file {
> +        "${users::builder::home}/.release-runner.ini":

This is kind of yucky.  I don't like daemons running out of /home, or production configuration hidden in dotfiles.  Not blocking this particular bug, but it'd sure be nice to clean that up sometime.

::: modules/releaserunner/manifests/settings.pp
@@ +5,5 @@
> +class releaserunner::settings {
> +    include ::config
> +    include users::builder
> +
> +    $root = "${users::builder::home}/release-runner"

Would moving this tool to /builds/releaserunner be as easy as changing this path?

Comment 12

[Rail Aliiev [:rail]]

Attachment: tools #2

don't hard code the config file location

Comment 13

[Rail Aliiev [:rail]]

Attachment: puppetize release-runner

* moved the root directory to /builds/releaserunner
* moved most of releaserunner config variables to moco-config.pp
* mercurial::repo requires $user now
* enabled stopasgroup/killasgroup

interdiff: https://gist.github.com/rail/5908786

Comment 14

[Dustin J. Mitchell [:dustin] (he/him)]

Comment on attachment 770132 [details] [diff] [review]
puppetize release-runner

Awesome!  r+ with two quick changes:

The config parameters still need to appear in modules/config/manifests/base.pp, just with empty values rather than moco-specific values.

It doesn't look like you've made the change to pass CONFIG to release-runner.sh, which from the previous attachment appears to be necessary.

Comment 15

[Rail Aliiev [:rail]]

Comment on attachment 770131 [details] [diff] [review]
tools #2

https://hg.mozilla.org/build/tools/rev/c9d59d2f344c
and deployed on bm36

Comment 16

[Rail Aliiev [:rail]]

Comment on attachment 770132 [details] [diff] [review]
puppetize release-runner

https://hg.mozilla.org/build/puppet/rev/310ffa3a58c3

Comment 17

[Rail Aliiev [:rail]]

and a bustage fix of course! https://hg.mozilla.org/build/puppet/rev/81e55c3be985

Comment 18

[Rail Aliiev [:rail]]

* Added 2 values to secrets.csv (the API credentials)
* Disabled release-runner on bm36:
  * stopped supervisor
  * commented out release runner section in supervisor.conf
  * mv ~cltbld/release-runner ~cltbld/release-runner-disabled
* Copied .ssh directory (keys) from bm36 to bm81

It's up and running!

TODO:
* add nagios checks for release-runner.py process
* monitor the next release

Comment 19

[Rail Aliiev [:rail]]

Another todo: https://wiki.mozilla.org/index.php?title=ReleaseEngineering/PuppetAgain/Modules/mercurial&action=edit&redlink=1

Comment 20

[Rail Aliiev [:rail]]

I had to remove mercurial-1.4-3.el6.x86_64 to make release-runner work.

Probably it would be better to explicitly pass path to hg (via config)

Comment 21

[Dustin J. Mitchell [:dustin] (he/him)]

It should probably always use the mozilla hg..

Comment 22

[Rail Aliiev [:rail]]

Attachment: venv: install mercurial

There is no easy way to specify path to hg to be used by hg modules. Installing mercurial into virtualenv will resolve the issue.

Comment 23

[Dustin J. Mitchell [:dustin] (he/him)]

oh, sorry, I thought you meant for mercurial::repo

Comment 24

[Rail Aliiev [:rail]]

Comment on attachment 770298 [details] [diff] [review]
venv: install mercurial

https://hg.mozilla.org/build/puppet/rev/da01cd9803a1

Comment 25

[Rail Aliiev [:rail]]

Callek suggested to avoid installing mercurial in virtualenv to avoid pains when we upgrade the mercurial version next time. We can alter PATH either in supervisor config (environment=PATH=/tools...:/bin:/usr/bin) or in release-runner.sh

Comment 26

[Rail Aliiev [:rail]]

Ben, do you have any strong opinion regarding using hg installed into a virtualenv vs the one we use for everything else (/tools/python/bin/python)? Currently we use http://hg.mozilla.org/build/puppet/file/7f8a1adc0e8c/modules/releaserunner/manifests/init.pp#l37

Comment 27

[[e-mail] (:bhearsum)]

(In reply to Rail Aliiev [:rail] from comment #26)
> Ben, do you have any strong opinion regarding using hg installed into a
> virtualenv vs the one we use for everything else (/tools/python/bin/python)?
> Currently we use
> http://hg.mozilla.org/build/puppet/file/7f8a1adc0e8c/modules/releaserunner/
> manifests/init.pp#l37

Seems a little silly to install a different hg for this, but I don't have strong feelings either way.

Comment 28

[Rail Aliiev [:rail]]

Attachment: use mozilla hg

Comment 29

[Rail Aliiev [:rail]]

Attachment: no mercurial in venv

Comment 30

[Justin Wood (:Callek)]

Comment on attachment 771385 [details] [diff] [review]
no mercurial in venv

Review of attachment 771385 [details] [diff] [review]:
-----------------------------------------------------------------

ahh wonderful

Comment 31

[[e-mail] (:bhearsum)]

Comment on attachment 771384 [details] [diff] [review]
use mozilla hg

Review of attachment 771384 [details] [diff] [review]:
-----------------------------------------------------------------

why is the supervisor file getting removed? because it moved to puppet?

Comment 32

[Rail Aliiev [:rail]]

(In reply to Ben Hearsum [:bhearsum] from comment #31)
> Comment on attachment 771384 [details] [diff] [review]
> use mozilla hg
> 
> Review of attachment 771384 [details] [diff] [review]:
> -----------------------------------------------------------------
> 
> why is the supervisor file getting removed? because it moved to puppet?

Yeah, that file isn't used anywhere, we added that just as an example.

Comment 33

[Rail Aliiev [:rail]]

Comment on attachment 771384 [details] [diff] [review]
use mozilla hg

https://hg.mozilla.org/build/tools/rev/fa6c6747e399 and deployed

Comment 34

[Rail Aliiev [:rail]]

Comment on attachment 771385 [details] [diff] [review]
no mercurial in venv

https://hg.mozilla.org/build/puppet/rev/b822974a36d1

Comment 35

[Rail Aliiev [:rail]]

[[e-mail] releaserunner]$ source /builds/releaserunner/bin/activate
(releaserunner)[[e-mail] releaserunner]$ pip uninstall mercurial
Uninstalling mercurial:
  /builds/releaserunner/bin/hg
  /builds/releaserunner/lib/python2.7/site-packages/hgext
  /builds/releaserunner/lib/python2.7/site-packages/mercurial
  /builds/releaserunner/lib/python2.7/site-packages/mercurial-2.5.4-py2.7.egg-info
Proceed (y/n)? y
  Successfully uninstalled mercurial