Deploy roller with releng puppet

RESOLVED FIXED

Status

RESOLVED FIXED
10 months ago
9 months ago

People

(Reporter: dividehex, Assigned: dividehex)

Tracking

(Blocks: 1 bug)

Details

Attachments

(6 attachments, 1 obsolete attachment)

(Assignee)

Description

10 months ago
The tentative plan is to manage deployment and provisioning of the roller service with releng puppet.  Since roller is built within docker images, I plan on using Ubuntu 16.04 (since this is our most modern OS supported) with a modern version of docker installed.  This also means mirroring the docker-ce apt repo to install docker.
(Assignee)

Comment 1

10 months ago
Created attachment 8941228 [details] [diff] [review]
minor fixes to ubuntu 16.04 puppet support

This:

1) allows puppet to realize cron as a systemd service
2) prevents the install iptables exec from running on every puppet run
Attachment #8941228 - Flags: review?(dhouse)

Updated

10 months ago
Attachment #8941228 - Flags: review?(dhouse) → review+
(Assignee)

Comment 3

10 months ago
Created attachment 8943396 [details] [diff] [review]
allow trusting of unsigned apt repos

I came across an issue with Ubuntu 16.04 where apt is not accepting unsigned repos.

For eg.

Reading package lists... Done
W: The repository 'https://puppetagain-apt.pvt.build.mozilla.org/repos/apt/custom/kernel xenial Release' does not have a Release file.
N: Data from such a repository can't be authenticated and is therefore potentially dangerous to use.
N: See apt-secure(8) manpage for repository creation and user configuration details.
W: The repository 'https://puppetagain-apt.pvt.build.mozilla.org/repos/apt/custom/mig-agent xenial Release' does not have a Release file.
N: Data from such a repository can't be authenticated and is therefore potentially dangerous to use.
N: See apt-secure(8) manpage for repository creation and user configuration details.


This patch allows for trusting of the custom (unsigned) repos.
Attachment #8943396 - Flags: review?(dhouse)

Updated

10 months ago
Attachment #8943396 - Flags: review?(dhouse) → review+
(Assignee)

Comment 5

10 months ago
Created attachment 8943455 [details] [diff] [review]
adjust apt source options and add docker-ce mirror repo

This adjusts the last patch to allow multiple options to be passed to the aptrepo defined resource and adds a docker-ce mirror repo as a virtual repo resource.
Attachment #8943455 - Flags: review?(dhouse)
(Assignee)

Comment 6

10 months ago
... and it looks like the fw module is going to conflict with the rules docker is putting in place. :-(

Notice: /Stage[main]/Main/Firewall[9009 fe610d70c21ce9c0931056b9ea87cf49]/ensure: removed
Notice: /Stage[main]/Main/Firewall[9003 efa39a4b89effc9642a89cf152ca1143]/ensure: removed
Notice: /Stage[main]/Main/Firewall[9002 f5c6b7c8832d4b28ed0fdf8a525e233e]/ensure: removed
Notice: /Stage[main]/Main/Firewall[9005 779207048a07114ad1f62eca677f85c7]/ensure: removed
Notice: /Stage[main]/Main/Firewall[9001 fec05d8f28ba51df24276694fc37936d]/ensure: removed
Notice: /Stage[main]/Main/Firewall[9012 e314954b40aa56dd6d50f3884f151c98]/ensure: removed
Notice: /Stage[main]/Main/Firewall[9011 4718f5ca335eeb57719ef8342fdfcd1b]/ensure: removed
Notice: /Stage[main]/Main/Firewall[9006 bbbb8cc8641f314ea26b871afd72a5e6]/ensure: removed
Notice: /Stage[main]/Main/Firewall[9004 cedf0366b942d83904bcc79dbe2bae22]/ensure: removed
Notice: /Stage[main]/Main/Firewall[9010 74aa613649d9718fb31361867b84366f]/ensure: removed
Notice: /Stage[main]/Main/Firewall[9008 305cfb035fc4adba0e46cad3d15bca23]/ensure: removed

Updated

10 months ago
Attachment #8943455 - Flags: review?(dhouse) → review+
(Assignee)

Comment 8

10 months ago
Created attachment 8943743 [details] [diff] [review]
purge firewall chains and ignore docker rules

The solution to the conflict between the puppet firewall module and dockers dynamic rules is to not purge ALL unmanaged rules but to only purge the 3 builtin chains (INPUT, OUTPUT and FORWARD) while ignoring rules matching docker.  Other user defined chains such as DOCKER, DOCKER-ISOLATION and DOCKER-USER are safely ignored also.
Attachment #8943743 - Flags: review?(dhouse)
(Assignee)

Updated

10 months ago
Attachment #8943743 - Attachment is patch: true

Updated

10 months ago
Attachment #8943743 - Flags: review?(dhouse) → review+
(Assignee)

Comment 9

10 months ago
Created attachment 8943746 [details] [diff] [review]
purge IPv4/IPv6 firewall chains and ignore docker rules

I forget to include the IPv6 chains.  This includes them.
Attachment #8943743 - Attachment is obsolete: true
Attachment #8943746 - Flags: review?(dhouse)
(Assignee)

Updated

10 months ago
Attachment #8943746 - Attachment is patch: true

Updated

10 months ago
Attachment #8943746 - Flags: review?(dhouse) → review+
(Assignee)

Comment 11

10 months ago
Created attachment 8947336 [details] [diff] [review]
setup and install roller

Roller puppet deployment

* Fixes puppet and docker iptables conflict
* adds puppet fw rules for roller
* adds Docker module for installing docker_ce and docker-compose
* adds systemd docker-compose services and cleanup timer
* adds roller module for installation and management

This is a fairly big puppet patch but it essentially adds all the bit needed to get roller installed and configured.  We will still need to fix up the .env files and to some extent the docker-compose.yml once we determine those values.
Attachment #8947336 - Flags: review?(dhouse)

Updated

10 months ago
Attachment #8947336 - Flags: review?(dhouse) → review+

Comment 13

9 months ago
Created attachment 8953177 [details] [diff] [review]
update puppet for using secrets and worker_config.json
Attachment #8953177 - Flags: review?(jwatkins)
(Assignee)

Comment 14

9 months ago
Comment on attachment 8953177 [details] [diff] [review]
update puppet for using secrets and worker_config.json

Review of attachment 8953177 [details] [diff] [review]:
-----------------------------------------------------------------

Looks good to me! r+
Attachment #8953177 - Flags: review?(jwatkins) → review+

Comment 15

9 months ago
Comment on attachment 8953177 [details] [diff] [review]
update puppet for using secrets and worker_config.json

Review of attachment 8953177 [details] [diff] [review]:
-----------------------------------------------------------------

::: modules/roller/manifests/systemd.pp
@@ +23,5 @@
>          "/etc/docker/compose/roller${environment}":
>              ensure => directory;
> +        "/etc/docker/compose/roller${environment}/worker_config.json":
> +            ensure  => file,
> +            content => template("roller/${environment}/worker_config.json.erb");

I'd like to make dev and prod use the same template instead of having ${environment} here
```
content => template("roller/worker_config.json.erb");
```
Attachment #8953177 - Flags: feedback?(jwatkins)
(Assignee)

Comment 16

9 months ago
(In reply to Dave House [:dhouse] from comment #15)
> Comment on attachment 8953177 [details] [diff] [review]
> update puppet for using secrets and worker_config.json
> 
> Review of attachment 8953177 [details] [diff] [review]:
> -----------------------------------------------------------------
> 
> ::: modules/roller/manifests/systemd.pp
> @@ +23,5 @@
> >          "/etc/docker/compose/roller${environment}":
> >              ensure => directory;
> > +        "/etc/docker/compose/roller${environment}/worker_config.json":
> > +            ensure  => file,
> > +            content => template("roller/${environment}/worker_config.json.erb");
> 
> I'd like to make dev and prod use the same template instead of having
> ${environment} here
> ```
> content => template("roller/worker_config.json.erb");
> ```

I'd prefer to keep the environments fairly independent of each other so we can test big changes on dev before pushing to prod.  We also want to limit the scope of hosts that dev can manage in order to minimize the impact if something goes wrong on the dev host.
(Assignee)

Updated

9 months ago
Attachment #8953177 - Flags: feedback?(jwatkins) → feedback-

Comment 17

9 months ago
Comment on attachment 8953177 [details] [diff] [review]
update puppet for using secrets and worker_config.json

remote:   https://hg.mozilla.org/build/puppet/rev/e2363817fa5bf5dde9c4138a8ecd3a0255de4f3f
Found mistake (left build entry for prod) and fixed with r=bustage.
```
# HG changeset patch
# User Dave House <dhouse@mozilla.com>
# Date 1519693280 25200
#      Mon Feb 26 18:01:20 2018 -0700
# Node ID 810c07d9b635dbeddd585a6ebe5ddc25e71956bb
# Parent  e2363817fa5bf5dde9c4138a8ecd3a0255de4f3f
Bug 1429240 - Roller. remove build from prod. r=bustage

diff --git a/modules/roller/templates/prod/docker-compose.yml.erb b/modules/roller/templates/prod/docker-compose.yml.erb
--- a/modules/roller/templates/prod/docker-compose.yml.erb
+++ b/modules/roller/templates/prod/docker-compose.yml.erb
@@ -18,8 +18,6 @@ services:

   worker:
     image: "mozilla/relops-hardware-controller:<%= @image_tag %>"
-    build:
-      context: /opt/rollerdev
     environment:
       - WORKER_CONFIG_PATH=/run/worker_config.json
     env_file:
```
remote:   https://hg.mozilla.org/build/puppet/rev/810c07d9b635dbeddd585a6ebe5ddc25e71956bb
Travis passed. pushing to production (tested on default from my env again):
Attachment #8953177 - Flags: checked-in+

Comment 18

9 months ago
Comment on attachment 8953177 [details] [diff] [review]
update puppet for using secrets and worker_config.json

Production push:
remote:   https://hg.mozilla.org/build/puppet/rev/024fd4af2d2e232d14549193731ee6836d5a8ba7

Updated

9 months ago
Status: NEW → RESOLVED
Last Resolved: 9 months ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.