Add puppet support for binary transparency workers

RESOLVED FIXED

Status

Release Engineering
Release Automation
P1
normal
RESOLVED FIXED
6 months ago
2 months ago

People

(Reporter: nthomas, Assigned: btang)

Tracking

(Blocks: 1 bug)

Firefox Tracking Flags

(Not tracked)

Details

Attachments

(2 attachments)

(Reporter)

Description

6 months ago
We can base this on https://hg.mozilla.org/build/puppet/file/production/modules/beetmover_scriptworker/, but will also need to install lego.
(Reporter)

Updated

6 months ago
Depends on: 1374964
(Assignee)

Comment 1

5 months ago
Created attachment 8885391 [details] [diff] [review]
diff for changes on build-puppet for transparencyscriptworker

- transparency scriptworker node was added to moco-nodes.pp
- new lego spec file was created
- 'realize' command in lego.pp is required for yum to install lego
- lego url path was added, repoflag number was increased in setup.pp
- transparencyscript version number updates in init.pp every time a new package of the transparencyscript is uploaded
- script_config.json and passwords.json files are initialized in init.pp using json templates that were created
- settings.pp includes variables used in template json files
Attachment #8885391 - Flags: review?(nthomas)
(Reporter)

Comment 2

5 months ago
Comment on attachment 8885391 [details] [diff] [review]
diff for changes on build-puppet for transparencyscriptworker

Review of attachment 8885391 [details] [diff] [review]:
-----------------------------------------------------------------

This is almost ready to go, just a few things to polish up. Please make a new patch with the suggestions, attach it to the bug, and request review.

::: manifests/moco-nodes.pp
@@ +1005,5 @@
>      include toplevel::server::pushapkscriptworker
>  }
>  
> +# Transparency scriptworkers
> +node /dev-linux64-ec2-btang\.dev\.releng.use1\.mozilla\.com/ {

Lets change this for production to:
node /binarytransparencyworker-.*\.srv\.releng\..*\.mozilla\.com/

@@ +1011,5 @@
> +    $transparencyworker_env = "dev"
> +    $timezone = "UTC"
> +    $only_user_ssh = true
> +    $pin_puppet_server = "releng-puppet2.srv.releng.scl3.mozilla.com"
> +    $pin_puppet_env = "btang"

Lets remove these two pinning definitions; we only use that during puppet development.

::: modules/packages/manifests/mozilla/lego.spec
@@ +17,5 @@
> +export GOPATH=%{buildroot}/go
> +mkdir bin
> +export GOBIN=$GOPATH/bin
> +go get -v
> +env GOOS=linux GOARCH=amd64 go build+

The + on the end of the line is a typo ?

::: modules/transparency_scriptworker/manifests/init.pp
@@ +86,5 @@
> +            mode      => '0600',
> +            owner     => $users::builder::username,
> +            group     => $users::builder::group,
> +            content   => template("${module_name}/script_config.json.erb"),
> +            show_diff => false;

We can remove the show_diff line now the credentials are in passwords.json.

::: modules/transparency_scriptworker/manifests/settings.pp
@@ +7,5 @@
> +    $task_script              = "${root}/bin/transparencyscript"
> +    $task_script_config       = "${root}/script_config.json"
> +    $task_max_timeout         = 1800
> +
> +    $worker_group             = 'test-dummy-workers'

The value here makes me think we should move this inside the the dev part of env_config, and update init.pp line 69.

::: modules/transparency_scriptworker/templates/script_config.json.erb
@@ +3,5 @@
> +
> +    "work_dir": "<%= scope.lookupvar("transparency_scriptworker::settings::root") %>/work",
> +    "public_artifact_dir": "<%= scope.lookupvar("transparency_scriptworker::settings::root") %>/artifacts/public",
> +    "aiohttp_max_connections": 10,
> +    "checksums_digests": ["sha512", "sha256"],

Lets remove aiohttp_max_connections and checksums_digests - looks like they're leftover from beetmover.

@@ +7,5 @@
> +    "checksums_digests": ["sha512", "sha256"],
> +
> +    "lego-path": "<%= scope.lookupvar("transparency_scriptworker::settings::lego_path") %>",
> +
> +    "schema_file": "<%= scope.lookupvar("transparency_scriptworker::settings::root") %>/lib/python3.5/site-packages/transparencyscript/data/transparency_task_schema.json",

We can remove schema_file too.

@@ +11,5 @@
> +    "schema_file": "<%= scope.lookupvar("transparency_scriptworker::settings::root") %>/lib/python3.5/site-packages/transparencyscript/data/transparency_task_schema.json",
> +
> +    "verbose": <%= scope.lookupvar("transparency_scriptworker::settings::verbose_logging") %>,
> +    "dummy": false,
> +    "disable_certs": false,

Don't think dummy or disable_certs do anything either, lets remove them.

In a JSON file, you're not allowed to have a trailing comma on the line before the closing }, so be sure to remove the comma from the verbose line.
Attachment #8885391 - Flags: review?(nthomas) → review-
(Assignee)

Comment 3

5 months ago
Created attachment 8885805 [details] [diff] [review]
bug1374963-v2.diff

- changes done according to first patch review
Attachment #8885805 - Flags: review?(nthomas)
(Reporter)

Comment 4

5 months ago
Comment on attachment 8885805 [details] [diff] [review]
bug1374963-v2.diff

Review of attachment 8885805 [details] [diff] [review]:
-----------------------------------------------------------------

Looks great r+. I also ran it through travis to check for linting issues and it came out clean - https://travis-ci.org/mozilla/build-puppet/builds/252988204.
Attachment #8885805 - Flags: review?(nthomas) → review+
(Reporter)

Comment 5

5 months ago
Comment on attachment 8885805 [details] [diff] [review]
bug1374963-v2.diff

default: https://hg.mozilla.org/build/puppet/rev/f32f75a79c630266006e7cf6b3c86359f2944f36
production: https://hg.mozilla.org/build/puppet/rev/80b0ae1d06910dbce37376a723ed11fc427404ff
Attachment #8885805 - Flags: checked-in+
(Reporter)

Comment 6

5 months ago
Lets do any follow up work in new bugs and close this one FIXED. \o/
Status: NEW → RESOLVED
Last Resolved: 5 months ago
Resolution: --- → FIXED
(Reporter)

Comment 7

5 months ago
Also copied the hiera secrets over from btang's environment to the main file.
(Reporter)

Updated

5 months ago
Blocks: 1380548
(Assignee)

Updated

5 months ago
Blocks: 1380829
(Reporter)

Updated

2 months ago
See Also: → bug 1404739
You need to log in before you can comment on or make changes to this bug.