1429240 - Deploy roller with releng puppet

Status: RESOLVED FIXED

(Infrastructure & Operations :: RelOps: Puppet, task)

Description

[Jake Watkins [:dividehex]]

The tentative plan is to manage deployment and provisioning of the roller service with releng puppet.  Since roller is built within docker images, I plan on using Ubuntu 16.04 (since this is our most modern OS supported) with a modern version of docker installed.  This also means mirroring the docker-ce apt repo to install docker.

Comment 1

[Jake Watkins [:dividehex]]

Attachment: minor fixes to ubuntu 16.04 puppet support

This:

1) allows puppet to realize cron as a systemd service
2) prevents the install iptables exec from running on every puppet run

Comment 2

[Jake Watkins [:dividehex]]

Comment on attachment 8941228 [details] [diff] [review]
minor fixes to ubuntu 16.04 puppet support

https://hg.mozilla.org/build/puppet/rev/34db8a805d90aadbaeb0931ca1c549260042923a
https://hg.mozilla.org/build/puppet/rev/431798fb0b43dcabe9477a63a8938bf07910cf3c

Comment 3

[Jake Watkins [:dividehex]]

Attachment: allow trusting of unsigned apt repos

I came across an issue with Ubuntu 16.04 where apt is not accepting unsigned repos.

For eg.

Reading package lists... Done
W: The repository 'https://puppetagain-apt.pvt.build.mozilla.org/repos/apt/custom/kernel xenial Release' does not have a Release file.
N: Data from such a repository can't be authenticated and is therefore potentially dangerous to use.
N: See apt-secure(8) manpage for repository creation and user configuration details.
W: The repository 'https://puppetagain-apt.pvt.build.mozilla.org/repos/apt/custom/mig-agent xenial Release' does not have a Release file.
N: Data from such a repository can't be authenticated and is therefore potentially dangerous to use.
N: See apt-secure(8) manpage for repository creation and user configuration details.


This patch allows for trusting of the custom (unsigned) repos.

Comment 4

[Jake Watkins [:dividehex]]

Comment on attachment 8943396 [details] [diff] [review]
allow trusting of unsigned apt repos

https://hg.mozilla.org/build/puppet/rev/4485d5d279915c31b4df2bdd138e82793c98297c
https://hg.mozilla.org/build/puppet/rev/771323337699766ba3070ab17567d37de741bac2

Comment 5

[Jake Watkins [:dividehex]]

Attachment: adjust apt source options and add docker-ce mirror repo

This adjusts the last patch to allow multiple options to be passed to the aptrepo defined resource and adds a docker-ce mirror repo as a virtual repo resource.

Comment 6

[Jake Watkins [:dividehex]]

... and it looks like the fw module is going to conflict with the rules docker is putting in place. :-(

Notice: /Stage[main]/Main/Firewall[9009 fe610d70c21ce9c0931056b9ea87cf49]/ensure: removed
Notice: /Stage[main]/Main/Firewall[9003 efa39a4b89effc9642a89cf152ca1143]/ensure: removed
Notice: /Stage[main]/Main/Firewall[9002 f5c6b7c8832d4b28ed0fdf8a525e233e]/ensure: removed
Notice: /Stage[main]/Main/Firewall[9005 779207048a07114ad1f62eca677f85c7]/ensure: removed
Notice: /Stage[main]/Main/Firewall[9001 fec05d8f28ba51df24276694fc37936d]/ensure: removed
Notice: /Stage[main]/Main/Firewall[9012 e314954b40aa56dd6d50f3884f151c98]/ensure: removed
Notice: /Stage[main]/Main/Firewall[9011 4718f5ca335eeb57719ef8342fdfcd1b]/ensure: removed
Notice: /Stage[main]/Main/Firewall[9006 bbbb8cc8641f314ea26b871afd72a5e6]/ensure: removed
Notice: /Stage[main]/Main/Firewall[9004 cedf0366b942d83904bcc79dbe2bae22]/ensure: removed
Notice: /Stage[main]/Main/Firewall[9010 74aa613649d9718fb31361867b84366f]/ensure: removed
Notice: /Stage[main]/Main/Firewall[9008 305cfb035fc4adba0e46cad3d15bca23]/ensure: removed

Comment 7

[Jake Watkins [:dividehex]]

Comment on attachment 8943455 [details] [diff] [review]
adjust apt source options and add docker-ce mirror repo

https://hg.mozilla.org/build/puppet/rev/af3868a1a34729efb058f2ec7c9c97762ea0874b
https://hg.mozilla.org/build/puppet/rev/81112cfbf76cfb0508c40e73b0052bbfb74acfaf

Comment 8

[Jake Watkins [:dividehex]]

Attachment: purge firewall chains and ignore docker rules

The solution to the conflict between the puppet firewall module and dockers dynamic rules is to not purge ALL unmanaged rules but to only purge the 3 builtin chains (INPUT, OUTPUT and FORWARD) while ignoring rules matching docker.  Other user defined chains such as DOCKER, DOCKER-ISOLATION and DOCKER-USER are safely ignored also.

Comment 9

[Jake Watkins [:dividehex]]

Attachment: purge IPv4/IPv6 firewall chains and ignore docker rules

I forget to include the IPv6 chains.  This includes them.

Comment 10

[Jake Watkins [:dividehex]]

Comment on attachment 8943746 [details] [diff] [review]
purge IPv4/IPv6 firewall chains and ignore docker rules

https://hg.mozilla.org/build/puppet/rev/e9ae4dcc28b863c2ae0ab79fd9278840ca5ac693
https://hg.mozilla.org/build/puppet/rev/9c371a6a31ac95101f93d6da12570549a2bf94d1

Comment 11

[Jake Watkins [:dividehex]]

Attachment: setup and install roller

Roller puppet deployment

* Fixes puppet and docker iptables conflict
* adds puppet fw rules for roller
* adds Docker module for installing docker_ce and docker-compose
* adds systemd docker-compose services and cleanup timer
* adds roller module for installation and management

This is a fairly big puppet patch but it essentially adds all the bit needed to get roller installed and configured.  We will still need to fix up the .env files and to some extent the docker-compose.yml once we determine those values.

Comment 12

[Jake Watkins [:dividehex]]

Comment on attachment 8947336 [details] [diff] [review]
setup and install roller

URL to git repo fixed.

Initial push to default:
https://hg.mozilla.org/build/puppet/rev/925b56f3ec8ca2283de728a953af74430a66f540

Puppet-lint fixes to default:
https://hg.mozilla.org/build/puppet/rev/8ef0bc28985d3eedaf6fa6275f25a77288c42e02

Merge default -> prod
https://hg.mozilla.org/build/puppet/rev/bbfa77f50b33a71483e14496eba254613d82b773

Comment 13

[:dhouse]

Attachment: update puppet for using secrets and worker_config.json

Comment 14

[Jake Watkins [:dividehex]]

Comment on attachment 8953177 [details] [diff] [review]
update puppet for using secrets and worker_config.json

Review of attachment 8953177 [details] [diff] [review]:
-----------------------------------------------------------------

Looks good to me! r+

Comment 15

[:dhouse]

Comment on attachment 8953177 [details] [diff] [review]
update puppet for using secrets and worker_config.json

Review of attachment 8953177 [details] [diff] [review]:
-----------------------------------------------------------------

::: modules/roller/manifests/systemd.pp
@@ +23,5 @@
>          "/etc/docker/compose/roller${environment}":
>              ensure => directory;
> +        "/etc/docker/compose/roller${environment}/worker_config.json":
> +            ensure  => file,
> +            content => template("roller/${environment}/worker_config.json.erb");

I'd like to make dev and prod use the same template instead of having ${environment} here
```
content => template("roller/worker_config.json.erb");
```

Comment 16

[Jake Watkins [:dividehex]]

(In reply to Dave House [:dhouse] from comment #15)
> Comment on attachment 8953177 [details] [diff] [review]
> update puppet for using secrets and worker_config.json
> 
> Review of attachment 8953177 [details] [diff] [review]:
> -----------------------------------------------------------------
> 
> ::: modules/roller/manifests/systemd.pp
> @@ +23,5 @@
> >          "/etc/docker/compose/roller${environment}":
> >              ensure => directory;
> > +        "/etc/docker/compose/roller${environment}/worker_config.json":
> > +            ensure  => file,
> > +            content => template("roller/${environment}/worker_config.json.erb");
> 
> I'd like to make dev and prod use the same template instead of having
> ${environment} here
> ```
> content => template("roller/worker_config.json.erb");
> ```

I'd prefer to keep the environments fairly independent of each other so we can test big changes on dev before pushing to prod.  We also want to limit the scope of hosts that dev can manage in order to minimize the impact if something goes wrong on the dev host.

Comment 17

[:dhouse]

Comment on attachment 8953177 [details] [diff] [review]
update puppet for using secrets and worker_config.json

remote:   https://hg.mozilla.org/build/puppet/rev/e2363817fa5bf5dde9c4138a8ecd3a0255de4f3f
Found mistake (left build entry for prod) and fixed with r=bustage.
```
# HG changeset patch
# User Dave House <[e-mail]>
# Date 1519693280 25200
#      Mon Feb 26 18:01:20 2018 -0700
# Node ID 810c07d9b635dbeddd585a6ebe5ddc25e71956bb
# Parent  e2363817fa5bf5dde9c4138a8ecd3a0255de4f3f
Bug 1429240 - Roller. remove build from prod. r=bustage

diff --git a/modules/roller/templates/prod/docker-compose.yml.erb b/modules/roller/templates/prod/docker-compose.yml.erb
--- a/modules/roller/templates/prod/docker-compose.yml.erb
+++ b/modules/roller/templates/prod/docker-compose.yml.erb
@@ -18,8 +18,6 @@ services:

   worker:
     image: "mozilla/relops-hardware-controller:<%= @image_tag %>"
-    build:
-      context: /opt/rollerdev
     environment:
       - WORKER_CONFIG_PATH=/run/worker_config.json
     env_file:
```
remote:   https://hg.mozilla.org/build/puppet/rev/810c07d9b635dbeddd585a6ebe5ddc25e71956bb
Travis passed. pushing to production (tested on default from my env again):

Comment 18

[:dhouse]

Comment on attachment 8953177 [details] [diff] [review]
update puppet for using secrets and worker_config.json

Production push:
remote:   https://hg.mozilla.org/build/puppet/rev/024fd4af2d2e232d14549193731ee6836d5a8ba7