870414 - setup scheduler manifests in puppetAgain

Status: RESOLVED FIXED

(Release Engineering :: General, defect)

Description

[hwine]

The scheduler master (currently bm36) was a special case in the old puppet manifests, even before the addition of multiple masters on the box.

Making separate from bug 803823 to highlight the differences.

Comment 1

[hwine]

Note: bug 823948 just added a new service to the scheduler host. It would be ideal if that could get folded into the new puppet configs, but the service can easily be installed manually again.

Comment 2

[Rail Aliiev [:rail]]

Hal, is there anything left here? AFAIK we moved the scheduler masters to bm81.

Comment 3

[Rail Aliiev [:rail]]

Attachment: [puppet] selfserve-agent

One stop closer to the moment when we can kill bm36 with fire! :)

* I still need to add bouncer_check nagios plugin/check
* Not sure what is the best way to manage a hg checkout (buildapi)
* Not tested. At all.

Comment 4

[Rail Aliiev [:rail]]

Callek suggested to extend mercurial::repo so it keeps repo checkout up to date, something like:

define mercurial::repo($hg_repo, $dst_dir, $user, $branch="", $rev="", $auto_update=false)

Comment 5

[Dustin J. Mitchell [:dustin] (he/him)]

Comment on attachment 815894 [details] [diff] [review]
[puppet] selfserve-agent

Review of attachment 815894 [details] [diff] [review]:
-----------------------------------------------------------------

::: modules/config/manifests/base.pp
@@ +66,5 @@
>      $releaserunner_sendchange_master = ""
>      $releaserunner_ssh_username = ""
>      $releaserunner_ssh_key = ""
>      $install_avds = "no"
> +    $selfserve_agent_root = "/builds/selfserve-agent"

Does this really need a config argument?  Particularly since the manifest invokes dirs::builds, which basically assumes it's in /builds.

@@ +67,5 @@
>      $releaserunner_ssh_username = ""
>      $releaserunner_ssh_key = ""
>      $install_avds = "no"
> +    $selfserve_agent_root = "/builds/selfserve-agent"
> +    $selfserve_agent_buildapi = "http://hg.mozilla.org/build/buildapi"

This should probably be $selfserve_agent_buildapi_repo

@@ +71,5 @@
> +    $selfserve_agent_buildapi = "http://hg.mozilla.org/build/buildapi"
> +    $selfserve_agent_buildapi_branch = "default"
> +    $selfserve_agent_sendchange_master = "bm81-build_scheduler"
> +    $selfserve_agent_branches = "http://hg.mozilla.org/build/tools/raw-file/default/buildfarm/maintenance/production-branches.json"
> +    $selfserve_agent_masters = "http://hg.mozilla.org/build/tools/raw-file/default/buildfarm/maintenance/production-masters.json"

These should both be $selfserve_agent_..._json

@@ +73,5 @@
> +    $selfserve_agent_sendchange_master = "bm81-build_scheduler"
> +    $selfserve_agent_branches = "http://hg.mozilla.org/build/tools/raw-file/default/buildfarm/maintenance/production-branches.json"
> +    $selfserve_agent_masters = "http://hg.mozilla.org/build/tools/raw-file/default/buildfarm/maintenance/production-masters.json"
> +    $selfserve_agent_clobberer = "http://clobberer.pvt.build.mozilla.org/index.php"
> +    $selfserve_agent_carrot_hostname = "rabbit2.build.scl1.mozilla.com"

/me hates on carrot for its SPOFfiness :(

@@ +77,5 @@
> +    $selfserve_agent_carrot_hostname = "rabbit2.build.scl1.mozilla.com"
> +    $selfserve_agent_carrot_vhost = "/buildapi"
> +    $selfserve_agent_carrot_userid = "buildapi"
> +    $selfserve_agent_carrot_exchange = "buildapi.control"
> +    $selfserve_agent_carrot_queue = "buildapi-agent-rabbit2"

That's a lot of config options to document :)

::: modules/selfserve_agent/manifests/init.pp
@@ +46,5 @@
> +            creates => "${selfserve_agent::settings::root}/lib/${packages::mozilla::python27::libdirname}/site-packages/buildapi.egg-link",
> +            user    => "${users::builder::username}",
> +            require => [Python::Virtualenv["${selfserve_agent::settings::root}"],
> +                        Mercurial::Repo["selfserve_agent-buildapi"]];
> +    }

Ewww, yuck!  The best choice here will be to make a buildapi package and put it in the puppetagain python repo.  When an upgrade needs to happen, put the new version in there and change the version string in the python::virtualenv.  This is what we're doing with mozpool, slaveapi, and a few others.  I see the automatic deployment from the branch in services.pp, though.  If that's really so common that you don't want to do new releases of buildapi for every change, then I'd rather see python::virtualenv extended to handle this case, maybe with editable => [ "/path/to/package/dir" ].  It can probably use the existing pip_check.py to determine whether the install is complete.

::: modules/selfserve_agent/manifests/services.pp
@@ +28,5 @@
> +            extra_config => template("${module_name}/extra_config.erb")
> +    }
> +
> +    exec {
> +        # TODO: maybe use cronjob instead?

Yes please :)

Comment 6

[Rail Aliiev [:rail]]

Attachment: [buildapi] release alias

"python setup.py release sdist" creates a tarball without "dev" suffix now

Comment 7

[Rail Aliiev [:rail]]

Comment on attachment 825885 [details] [diff] [review]
[buildapi] release alias

https://hg.mozilla.org/build/buildapi/rev/7e02b276acf4

Comment 8

[Rail Aliiev [:rail]]

Attachment: [buildapi] add console_scripts entry points

Dumping my current status.

* Need to include buildapi/scripts into tarballs (added to MANIFEST.in)
* add console_scripts for selfserve-agent to simplify puppet paths
** it requires renaming selfserve-agent.py to selfserve_agent.py and adding main(). For backward compatibility done by symlinking.

Comment 9

[Rail Aliiev [:rail]]

Attachment: [puppet] selfserve-agent

No more hacky modules using a mercurial checkout.

Comment 10

[Rail Aliiev [:rail]]

Attachment: [puppet] selfserve-agent

I tested this one and managed to install selfserve-agent. I had to add a lot of dependencies, because otherwise the script refuses to run. :/

I left config::selfserve_agent_root as is to make the selfserve-agent.ini.erb more consistent and we have the same pattern for release-runner.

I still need to package buildapi and requests netflows to rabbit2.build.scl1.mozilla.com from bm81

Comment 11

[Rail Aliiev [:rail]]

Attachment: [buildapi] add console_scripts entry points

forgot to add and empty __init__.py

Comment 12

[Dustin J. Mitchell [:dustin] (he/him)]

Comment on attachment 826790 [details] [diff] [review]
[puppet] selfserve-agent

Review of attachment 826790 [details] [diff] [review]:
-----------------------------------------------------------------

r+ with these minor changes

::: modules/config/manifests/base.pp
@@ +76,5 @@
> +    $selfserve_agent_carrot_hostname = "rabbit2.build.scl1.mozilla.com"
> +    $selfserve_agent_carrot_vhost = "/buildapi"
> +    $selfserve_agent_carrot_userid = "buildapi"
> +    $selfserve_agent_carrot_exchange = "buildapi.control"
> +    $selfserve_agent_carrot_queue = "buildapi-agent-rabbit2"

These settings (aside from the directory) should be in moco-nodes.pp, not in base.pp

::: modules/selfserve_agent/manifests/init.pp
@@ +55,5 @@
> +                "sqlalchemy-migrate==0.7.2",
> +                "wsgiref==0.1.2",
> +                "zope.interface==4.0.2",
> +            ];
> +    }

Does it make sense to move this to `selfserve_agent::install`?

::: modules/selfserve_agent/manifests/services.pp
@@ +2,5 @@
> +# License, v. 2.0. If a copy of the MPL was not distributed with this
> +# file, You can obtain one at http://mozilla.org/MPL/2.0/.
> +class selfserve_agent::services {
> +    include ::config
> +    include packages::mozilla::supervisor

the supervisord module should be including this, so no need to do so here

Comment 13

[Rail Aliiev [:rail]]

Attachment: [puppet] selfserve-agent

* addressed comments
* tested and seen a noop check
* carry over r=dustin

Comment 14

[Rail Aliiev [:rail]]

Comment on attachment 826793 [details] [diff] [review]
[buildapi] add console_scripts entry points

https://hg.mozilla.org/build/buildapi/rev/250614ab1b08

Comment 15

[Rail Aliiev [:rail]]

Attachment: [puppet] bouncer nagios check

This is the last piece of bm36 AFAIK.

nagios-tools's temporarily lives here: https://github.com/rail/build-nagios-tools/compare/packaging

I may change its name to something more distinguishable (names are welcome).

Comment 16

[Dustin J. Mitchell [:dustin] (he/him)]

Comment on attachment 826874 [details] [diff] [review]
[puppet] bouncer nagios check

The nrpe check is a new subsidiary role for this master, so it should be included in a module of its own (trivial though it is).  I know that the master stuff is a mess in puppet, with lots of detail in the node definitions, but we don't need to make it worse :)

Do we really need a full virtualenv just for a nagios check?  If so, and if that virtualenv is something more general than a bouncer check, I think that it should be installed by a separate module -- or maybe a class in the nrpe module -- and required by nrpe::check::bouncer.

Comment 17

[Rail Aliiev [:rail]]

Attachment: [puppet] bouncer nagios check

Something like this?

I still would like to install this check in a separate virtualenv for varous reasons:
* this is how it's installed on bm36
* it requires an external module (nagiosplugin)
* virtualenv is cheap! :)
* to keep the nagios script(s) isolated from other environments

Comment 18

[Rail Aliiev [:rail]]

Comment on attachment 826803 [details] [diff] [review]
[puppet] selfserve-agent

https://hg.mozilla.org/build/puppet/rev/5acc6e2544b9

Comment 19

[Rail Aliiev [:rail]]

Comment on attachment 826954 [details] [diff] [review]
[puppet] bouncer nagios check

This has been tested and worked fine.

Comment 20

[Rail Aliiev [:rail]]

Comment on attachment 826954 [details] [diff] [review]
[puppet] bouncer nagios check

https://hg.mozilla.org/build/puppet/rev/d340a4994244

Comment 21

[Rail Aliiev [:rail]]

Attachment: use $nrpe::settings::plugins_dir

Comment 22

[Rail Aliiev [:rail]]

Comment on attachment 830408 [details] [diff] [review]
use $nrpe::settings::plugins_dir

https://hg.mozilla.org/build/puppet/rev/b1a992ffcc4d

Comment 23

[Rail Aliiev [:rail]]

I think we are done here! \o/