1374963 - Add puppet support for binary transparency workers

Status: RESOLVED FIXED

(Release Engineering :: Release Automation: Other, enhancement, P1)

Description

[Nick Thomas [:nthomas] (UTC+12)]

We can base this on https://hg.mozilla.org/build/puppet/file/production/modules/beetmover_scriptworker/, but will also need to install lego.

Comment 1

[Brandon Tang [:btang]]

Attachment: diff for changes on build-puppet for transparencyscriptworker

- transparency scriptworker node was added to moco-nodes.pp
- new lego spec file was created
- 'realize' command in lego.pp is required for yum to install lego
- lego url path was added, repoflag number was increased in setup.pp
- transparencyscript version number updates in init.pp every time a new package of the transparencyscript is uploaded
- script_config.json and passwords.json files are initialized in init.pp using json templates that were created
- settings.pp includes variables used in template json files

Comment 2

[Nick Thomas [:nthomas] (UTC+12)]

Comment on attachment 8885391 [details] [diff] [review]
diff for changes on build-puppet for transparencyscriptworker

Review of attachment 8885391 [details] [diff] [review]:
-----------------------------------------------------------------

This is almost ready to go, just a few things to polish up. Please make a new patch with the suggestions, attach it to the bug, and request review.

::: manifests/moco-nodes.pp
@@ +1005,5 @@
>      include toplevel::server::pushapkscriptworker
>  }
>  
> +# Transparency scriptworkers
> +node /dev-linux64-ec2-btang\.dev\.releng.use1\.mozilla\.com/ {

Lets change this for production to:
node /binarytransparencyworker-.*\.srv\.releng\..*\.mozilla\.com/

@@ +1011,5 @@
> +    $transparencyworker_env = "dev"
> +    $timezone = "UTC"
> +    $only_user_ssh = true
> +    $pin_puppet_server = "releng-puppet2.srv.releng.scl3.mozilla.com"
> +    $pin_puppet_env = "btang"

Lets remove these two pinning definitions; we only use that during puppet development.

::: modules/packages/manifests/mozilla/lego.spec
@@ +17,5 @@
> +export GOPATH=%{buildroot}/go
> +mkdir bin
> +export GOBIN=$GOPATH/bin
> +go get -v
> +env GOOS=linux GOARCH=amd64 go build+

The + on the end of the line is a typo ?

::: modules/transparency_scriptworker/manifests/init.pp
@@ +86,5 @@
> +            mode      => '0600',
> +            owner     => $users::builder::username,
> +            group     => $users::builder::group,
> +            content   => template("${module_name}/script_config.json.erb"),
> +            show_diff => false;

We can remove the show_diff line now the credentials are in passwords.json.

::: modules/transparency_scriptworker/manifests/settings.pp
@@ +7,5 @@
> +    $task_script              = "${root}/bin/transparencyscript"
> +    $task_script_config       = "${root}/script_config.json"
> +    $task_max_timeout         = 1800
> +
> +    $worker_group             = 'test-dummy-workers'

The value here makes me think we should move this inside the the dev part of env_config, and update init.pp line 69.

::: modules/transparency_scriptworker/templates/script_config.json.erb
@@ +3,5 @@
> +
> +    "work_dir": "<%= scope.lookupvar("transparency_scriptworker::settings::root") %>/work",
> +    "public_artifact_dir": "<%= scope.lookupvar("transparency_scriptworker::settings::root") %>/artifacts/public",
> +    "aiohttp_max_connections": 10,
> +    "checksums_digests": ["sha512", "sha256"],

Lets remove aiohttp_max_connections and checksums_digests - looks like they're leftover from beetmover.

@@ +7,5 @@
> +    "checksums_digests": ["sha512", "sha256"],
> +
> +    "lego-path": "<%= scope.lookupvar("transparency_scriptworker::settings::lego_path") %>",
> +
> +    "schema_file": "<%= scope.lookupvar("transparency_scriptworker::settings::root") %>/lib/python3.5/site-packages/transparencyscript/data/transparency_task_schema.json",

We can remove schema_file too.

@@ +11,5 @@
> +    "schema_file": "<%= scope.lookupvar("transparency_scriptworker::settings::root") %>/lib/python3.5/site-packages/transparencyscript/data/transparency_task_schema.json",
> +
> +    "verbose": <%= scope.lookupvar("transparency_scriptworker::settings::verbose_logging") %>,
> +    "dummy": false,
> +    "disable_certs": false,

Don't think dummy or disable_certs do anything either, lets remove them.

In a JSON file, you're not allowed to have a trailing comma on the line before the closing }, so be sure to remove the comma from the verbose line.

Comment 3

[Brandon Tang [:btang]]

Attachment: bug1374963-v2.diff

- changes done according to first patch review

Comment 4

[Nick Thomas [:nthomas] (UTC+12)]

Comment on attachment 8885805 [details] [diff] [review]
bug1374963-v2.diff

Review of attachment 8885805 [details] [diff] [review]:
-----------------------------------------------------------------

Looks great r+. I also ran it through travis to check for linting issues and it came out clean - https://travis-ci.org/mozilla/build-puppet/builds/252988204.

Comment 5

[Nick Thomas [:nthomas] (UTC+12)]

Comment on attachment 8885805 [details] [diff] [review]
bug1374963-v2.diff

default: https://hg.mozilla.org/build/puppet/rev/f32f75a79c630266006e7cf6b3c86359f2944f36
production: https://hg.mozilla.org/build/puppet/rev/80b0ae1d06910dbce37376a723ed11fc427404ff

Comment 6

[Nick Thomas [:nthomas] (UTC+12)]

Lets do any follow up work in new bugs and close this one FIXED. \o/

Comment 7

[Nick Thomas [:nthomas] (UTC+12)]

Also copied the hiera secrets over from btang's environment to the main file.