Closed Bug 1429240 Opened 6 years ago Closed 6 years ago

Deploy roller with releng puppet

Categories

(Infrastructure & Operations :: RelOps: Puppet, task)

task
Not set
normal

Tracking

(Not tracked)

RESOLVED FIXED

People

(Reporter: dividehex, Assigned: dividehex)

References

Details

Attachments

(6 files, 1 obsolete file)

The tentative plan is to manage deployment and provisioning of the roller service with releng puppet.  Since roller is built within docker images, I plan on using Ubuntu 16.04 (since this is our most modern OS supported) with a modern version of docker installed.  This also means mirroring the docker-ce apt repo to install docker.
This:

1) allows puppet to realize cron as a systemd service
2) prevents the install iptables exec from running on every puppet run
Attachment #8941228 - Flags: review?(dhouse)
Attachment #8941228 - Flags: review?(dhouse) → review+
I came across an issue with Ubuntu 16.04 where apt is not accepting unsigned repos.

For eg.

Reading package lists... Done
W: The repository 'https://puppetagain-apt.pvt.build.mozilla.org/repos/apt/custom/kernel xenial Release' does not have a Release file.
N: Data from such a repository can't be authenticated and is therefore potentially dangerous to use.
N: See apt-secure(8) manpage for repository creation and user configuration details.
W: The repository 'https://puppetagain-apt.pvt.build.mozilla.org/repos/apt/custom/mig-agent xenial Release' does not have a Release file.
N: Data from such a repository can't be authenticated and is therefore potentially dangerous to use.
N: See apt-secure(8) manpage for repository creation and user configuration details.


This patch allows for trusting of the custom (unsigned) repos.
Attachment #8943396 - Flags: review?(dhouse)
Attachment #8943396 - Flags: review?(dhouse) → review+
This adjusts the last patch to allow multiple options to be passed to the aptrepo defined resource and adds a docker-ce mirror repo as a virtual repo resource.
Attachment #8943455 - Flags: review?(dhouse)
... and it looks like the fw module is going to conflict with the rules docker is putting in place. :-(

Notice: /Stage[main]/Main/Firewall[9009 fe610d70c21ce9c0931056b9ea87cf49]/ensure: removed
Notice: /Stage[main]/Main/Firewall[9003 efa39a4b89effc9642a89cf152ca1143]/ensure: removed
Notice: /Stage[main]/Main/Firewall[9002 f5c6b7c8832d4b28ed0fdf8a525e233e]/ensure: removed
Notice: /Stage[main]/Main/Firewall[9005 779207048a07114ad1f62eca677f85c7]/ensure: removed
Notice: /Stage[main]/Main/Firewall[9001 fec05d8f28ba51df24276694fc37936d]/ensure: removed
Notice: /Stage[main]/Main/Firewall[9012 e314954b40aa56dd6d50f3884f151c98]/ensure: removed
Notice: /Stage[main]/Main/Firewall[9011 4718f5ca335eeb57719ef8342fdfcd1b]/ensure: removed
Notice: /Stage[main]/Main/Firewall[9006 bbbb8cc8641f314ea26b871afd72a5e6]/ensure: removed
Notice: /Stage[main]/Main/Firewall[9004 cedf0366b942d83904bcc79dbe2bae22]/ensure: removed
Notice: /Stage[main]/Main/Firewall[9010 74aa613649d9718fb31361867b84366f]/ensure: removed
Notice: /Stage[main]/Main/Firewall[9008 305cfb035fc4adba0e46cad3d15bca23]/ensure: removed
Attachment #8943455 - Flags: review?(dhouse) → review+
The solution to the conflict between the puppet firewall module and dockers dynamic rules is to not purge ALL unmanaged rules but to only purge the 3 builtin chains (INPUT, OUTPUT and FORWARD) while ignoring rules matching docker.  Other user defined chains such as DOCKER, DOCKER-ISOLATION and DOCKER-USER are safely ignored also.
Attachment #8943743 - Flags: review?(dhouse)
Attachment #8943743 - Attachment is patch: true
Attachment #8943743 - Flags: review?(dhouse) → review+
I forget to include the IPv6 chains.  This includes them.
Attachment #8943743 - Attachment is obsolete: true
Attachment #8943746 - Flags: review?(dhouse)
Attachment #8943746 - Attachment is patch: true
Attachment #8943746 - Flags: review?(dhouse) → review+
Roller puppet deployment

* Fixes puppet and docker iptables conflict
* adds puppet fw rules for roller
* adds Docker module for installing docker_ce and docker-compose
* adds systemd docker-compose services and cleanup timer
* adds roller module for installation and management

This is a fairly big puppet patch but it essentially adds all the bit needed to get roller installed and configured.  We will still need to fix up the .env files and to some extent the docker-compose.yml once we determine those values.
Attachment #8947336 - Flags: review?(dhouse)
Attachment #8947336 - Flags: review?(dhouse) → review+
Comment on attachment 8953177 [details] [diff] [review]
update puppet for using secrets and worker_config.json

Review of attachment 8953177 [details] [diff] [review]:
-----------------------------------------------------------------

Looks good to me! r+
Attachment #8953177 - Flags: review?(jwatkins) → review+
Comment on attachment 8953177 [details] [diff] [review]
update puppet for using secrets and worker_config.json

Review of attachment 8953177 [details] [diff] [review]:
-----------------------------------------------------------------

::: modules/roller/manifests/systemd.pp
@@ +23,5 @@
>          "/etc/docker/compose/roller${environment}":
>              ensure => directory;
> +        "/etc/docker/compose/roller${environment}/worker_config.json":
> +            ensure  => file,
> +            content => template("roller/${environment}/worker_config.json.erb");

I'd like to make dev and prod use the same template instead of having ${environment} here
```
content => template("roller/worker_config.json.erb");
```
Attachment #8953177 - Flags: feedback?(jwatkins)
(In reply to Dave House [:dhouse] from comment #15)
> Comment on attachment 8953177 [details] [diff] [review]
> update puppet for using secrets and worker_config.json
> 
> Review of attachment 8953177 [details] [diff] [review]:
> -----------------------------------------------------------------
> 
> ::: modules/roller/manifests/systemd.pp
> @@ +23,5 @@
> >          "/etc/docker/compose/roller${environment}":
> >              ensure => directory;
> > +        "/etc/docker/compose/roller${environment}/worker_config.json":
> > +            ensure  => file,
> > +            content => template("roller/${environment}/worker_config.json.erb");
> 
> I'd like to make dev and prod use the same template instead of having
> ${environment} here
> ```
> content => template("roller/worker_config.json.erb");
> ```

I'd prefer to keep the environments fairly independent of each other so we can test big changes on dev before pushing to prod.  We also want to limit the scope of hosts that dev can manage in order to minimize the impact if something goes wrong on the dev host.
Attachment #8953177 - Flags: feedback?(jwatkins) → feedback-
Comment on attachment 8953177 [details] [diff] [review]
update puppet for using secrets and worker_config.json

remote:   https://hg.mozilla.org/build/puppet/rev/e2363817fa5bf5dde9c4138a8ecd3a0255de4f3f
Found mistake (left build entry for prod) and fixed with r=bustage.
```
# HG changeset patch
# User Dave House <dhouse@mozilla.com>
# Date 1519693280 25200
#      Mon Feb 26 18:01:20 2018 -0700
# Node ID 810c07d9b635dbeddd585a6ebe5ddc25e71956bb
# Parent  e2363817fa5bf5dde9c4138a8ecd3a0255de4f3f
Bug 1429240 - Roller. remove build from prod. r=bustage

diff --git a/modules/roller/templates/prod/docker-compose.yml.erb b/modules/roller/templates/prod/docker-compose.yml.erb
--- a/modules/roller/templates/prod/docker-compose.yml.erb
+++ b/modules/roller/templates/prod/docker-compose.yml.erb
@@ -18,8 +18,6 @@ services:

   worker:
     image: "mozilla/relops-hardware-controller:<%= @image_tag %>"
-    build:
-      context: /opt/rollerdev
     environment:
       - WORKER_CONFIG_PATH=/run/worker_config.json
     env_file:
```
remote:   https://hg.mozilla.org/build/puppet/rev/810c07d9b635dbeddd585a6ebe5ddc25e71956bb
Travis passed. pushing to production (tested on default from my env again):
Attachment #8953177 - Flags: checked-in+
Comment on attachment 8953177 [details] [diff] [review]
update puppet for using secrets and worker_config.json

Production push:
remote:   https://hg.mozilla.org/build/puppet/rev/024fd4af2d2e232d14549193731ee6836d5a8ba7
Status: NEW → RESOLVED
Closed: 6 years ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Created:
Updated:
Size: