1409609 - set up dev beetmover, balrog scriptworker pools

Status: RESOLVED FIXED

(Release Engineering :: Release Automation: Other, enhancement)

Comment 1

[Aki Sasaki (not active)]

Attachment: [needs testing] beetmover-dev.patch

Comment 2

[Aki Sasaki (not active)]

My puppet env is ready to test beetmover-dev. I should whip up the other two patches... then spin up instances and puppetize.

Comment 3

[Aki Sasaki (not active)]

I think we can use breakpoint (non-existing workerTypes) for staging pushapk.

Comment 4

[Aki Sasaki (not active)]

Attachment: bug 1409609 - beetmover-dev.

Review commit: https://reviewboard.mozilla.org/r/190838/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/190838/

Comment 5

[Aki Sasaki (not active)]

Attachment: bug 1409609 - balrog-dev.

Review commit: https://reviewboard.mozilla.org/r/190840/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/190840/

Comment 6

[Mihai Tabara [:mtabara]⌚️GMT]

Comment on attachment 8919890 [details]
bug 1409609 - beetmover-dev.

https://reviewboard.mozilla.org/r/190838/#review196380

::: manifests/moco-nodes.pp:1012
(Diff revision 1)
>      $timezone            = 'UTC'
>      $only_user_ssh       = true
>      include toplevel::server::beetmoverscriptworker
>  }
>  
> +node /beetmover-dev.*\.releng\..*\.mozilla\.com/ {

hm, so the dev-beetmoverworkers won't have the "srv" within them?

::: modules/beetmover_scriptworker/manifests/settings.pp:33
(Diff revision 1)
>              dep_beetmover_aws_s3_firefox_bucket     => 'net-mozaws-stage-delivery-firefox',
>              dep_beetmover_aws_s3_fennec_bucket      => 'net-mozaws-stage-delivery-archive',
> +
> +            config_template                         => 'beetmover_scriptworker/dev_script_config.json.erb',
> +            worker_type                             => 'beetmover-dev',
> +            taskcluster_client_id                   => secret('beetmoverworker_dev_taskcluster_client_id'),

They seem to match the production credentials. Is this expected or did we not generate a separate set of credentials just yet?

::: modules/beetmover_scriptworker/templates/base_script_config.json.erb:37
(Diff revision 1)
>              "fennec_candidates": "<%= scope.lookupvar("beetmover_scriptworker::settings::root") %>/lib/python3.5/site-packages/beetmoverscript/templates/fennec_candidates.yml",
>              "fennec_candidates_repacks": "<%= scope.lookupvar("beetmover_scriptworker::settings::root") %>/lib/python3.5/site-packages/beetmoverscript/templates/fennec_candidates_repacks.yml",
>              "fennecx86_candidates": "<%= scope.lookupvar("beetmover_scriptworker::settings::root") %>/lib/python3.5/site-packages/beetmoverscript/templates/fennecx86_candidates.yml",
>              "fennecaarch64_candidates": "<%= scope.lookupvar("beetmover_scriptworker::settings::root") %>/lib/python3.5/site-packages/beetmoverscript/templates/fennecaarch64_candidates.yml"
>          },
> -        "push-to-releases": {},
> +        "push-to-releases": {}

I think we can trim this as well until we add beetmoverscript support(https://github.com/mozilla-releng/beetmoverscript/blob/master/beetmoverscript/script.py#L80) to replace the existing mozharness candidates => releases copyover script(https://hg.mozilla.org/mozilla-central/file/tip/testing/mozharness/scripts/release/push-candidate-to-releases.py)

Comment 7

[Mihai Tabara [:mtabara]⌚️GMT]

Comment on attachment 8919891 [details]
bug 1409609 - balrog-dev.

https://reviewboard.mozilla.org/r/190840/#review196386

::: modules/balrog_scriptworker/manifests/settings.pp:22
(Diff revision 1)
>              balrog_username => 'balrog-stage-ffxbld',
>              balrog_password => secret('balrog-stage-ffxbld_ldap_password'),
>              balrog_api_root => 'https://balrog-admin.stage.mozaws.net/api',
> +
> +            dummy => true,
> +            tools_repo => 'https://hg.mozilla.org/build/tools',

Questions: why is the tools repo per production/staging? So that we can point it to other tools repo if needed?

::: modules/balrog_scriptworker/manifests/settings.pp:23
(Diff revision 1)
>              balrog_password => secret('balrog-stage-ffxbld_ldap_password'),
>              balrog_api_root => 'https://balrog-admin.stage.mozaws.net/api',
> +
> +            dummy => true,
> +            tools_repo => 'https://hg.mozilla.org/build/tools',
> +            taskcluster_client_id => 'project/releng/scriptworker/balrogworker-dev',

Nit: for consistency across {}-scriptworkers, we should use the secret() function here as well, as we did for beetmoverworkers. Or switch in the latter to reflect how it is here.

::: modules/balrog_scriptworker/manifests/settings.pp:38
(Diff revision 1)
>              balrog_api_root => 'https://aus4-admin.mozilla.org/api',
> +
> +            dummy => false,
> +            tools_repo => 'https://hg.mozilla.org/build/tools',
> +            taskcluster_client_id => 'project/releng/scriptworker/balrogworker',
> +            taskcluster_access_token => secret('balrogworker_prod_taskcluster_access_token'),

Similar problem, dev vs prod credentials seem to match. I suppose this is expected since we didn't generate a new set but just thought it's worth raising.

::: modules/balrog_scriptworker/templates/script_config.json.erb:11
(Diff revision 1)
>  
>      "verbose": <%= scope.lookupvar("balrog_scriptworker::settings::verbose_logging") %>,
> -    "dummy": false,
> +    "dummy": @env_config["dummy"],
>      "disable_certs": false,
>  
>      "server_config": {

Question: I suppose dev vs production will be equally safe from a security perspective right? I mean, there's no reason to ease the security on the dev balrogworkers as opposed to the other one. Otherwise, I'm thinking we should split these configs somehow the same way as you did for beetmover since prod passwords would get this way on the dev-balrogworkers instances.

Comment 8

[Aki Sasaki (not active)]

Thank you for the reviews!

(In reply to Mihai Tabara [:mtabara]⌚️GMT from comment #6)
> > +node /beetmover-dev.*\.releng\..*\.mozilla\.com/ {
> 
> hm, so the dev-beetmoverworkers won't have the "srv" within them?

Good catch :)
It worked, because the .* greedy matched, but I'm adding the .srv back in.

> ::: modules/beetmover_scriptworker/manifests/settings.pp:33
> (Diff revision 1)
> >              dep_beetmover_aws_s3_firefox_bucket     => 'net-mozaws-stage-delivery-firefox',
> >              dep_beetmover_aws_s3_fennec_bucket      => 'net-mozaws-stage-delivery-archive',
> > +
> > +            config_template                         => 'beetmover_scriptworker/dev_script_config.json.erb',
> > +            worker_type                             => 'beetmover-dev',
> > +            taskcluster_client_id                   => secret('beetmoverworker_dev_taskcluster_client_id'),
> 
> They seem to match the production credentials. Is this expected or did we
> not generate a separate set of credentials just yet?

This is because the current prod secrets are called `beetmover_dev*`. Rather than create a new secret named `beetmover_dev_for_realz*` I copied the `beetmover_dev*` secrets to `beetmover_prod*`. I haven't replaced the `beetmover_dev*` secrets because without this patch, that would replace the current beetmover scriptworkers' secrets with the new dev secrets.

The new generated secrets are in my environment; beetmover-dev1 is pinned to that env.

> ::: modules/beetmover_scriptworker/templates/base_script_config.json.erb:37
> (Diff revision 1)
> >              "fennec_candidates": "<%= scope.lookupvar("beetmover_scriptworker::settings::root") %>/lib/python3.5/site-packages/beetmoverscript/templates/fennec_candidates.yml",
> >              "fennec_candidates_repacks": "<%= scope.lookupvar("beetmover_scriptworker::settings::root") %>/lib/python3.5/site-packages/beetmoverscript/templates/fennec_candidates_repacks.yml",
> >              "fennecx86_candidates": "<%= scope.lookupvar("beetmover_scriptworker::settings::root") %>/lib/python3.5/site-packages/beetmoverscript/templates/fennecx86_candidates.yml",
> >              "fennecaarch64_candidates": "<%= scope.lookupvar("beetmover_scriptworker::settings::root") %>/lib/python3.5/site-packages/beetmoverscript/templates/fennecaarch64_candidates.yml"
> >          },
> > -        "push-to-releases": {},
> > +        "push-to-releases": {}
> 
> I think we can trim this as well until we add beetmoverscript
> support(https://github.com/mozilla-releng/beetmoverscript/blob/master/
> beetmoverscript/script.py#L80) to replace the existing mozharness candidates
> => releases copyover
> script(https://hg.mozilla.org/mozilla-central/file/tip/testing/mozharness/
> scripts/release/push-candidate-to-releases.py)

I'm adding that now :)

Comment 9

[Aki Sasaki (not active)]

(In reply to Mihai Tabara [:mtabara]⌚️GMT from comment #7)
> Comment on attachment 8919891 [details]
> bug 1409609 - balrog-dev.
> 
> https://reviewboard.mozilla.org/r/190840/#review196386
> 
> ::: modules/balrog_scriptworker/manifests/settings.pp:22
> (Diff revision 1)
> >              balrog_username => 'balrog-stage-ffxbld',
> >              balrog_password => secret('balrog-stage-ffxbld_ldap_password'),
> >              balrog_api_root => 'https://balrog-admin.stage.mozaws.net/api',
> > +
> > +            dummy => true,
> > +            tools_repo => 'https://hg.mozilla.org/build/tools',
> 
> Questions: why is the tools repo per production/staging? So that we can
> point it to other tools repo if needed?

Yes. I debated doing this -- if we use an env, we can switch the shared tools repo without worry. But we might want to pin dev against a tools repo in prod puppet for some reason; this would allow for that.

> ::: modules/balrog_scriptworker/manifests/settings.pp:23
> (Diff revision 1)
> >              balrog_password => secret('balrog-stage-ffxbld_ldap_password'),
> >              balrog_api_root => 'https://balrog-admin.stage.mozaws.net/api',
> > +
> > +            dummy => true,
> > +            tools_repo => 'https://hg.mozilla.org/build/tools',
> > +            taskcluster_client_id => 'project/releng/scriptworker/balrogworker-dev',
> 
> Nit: for consistency across {}-scriptworkers, we should use the secret()
> function here as well, as we did for beetmoverworkers. Or switch in the
> latter to reflect how it is here.

I thought about it.

Arguments for cleartext client ids:
- IMO it's easier to slip up if everything's in hiera -- it's unclear what client id we're using, and we have to ssh in and decrypt it to figure it out.
  - similarly, it's hard to know which client to update if we want to adjust scopes.
- the client id isn't secret.
- fewer hiera edits

Arguments against:
- consistency
- ???

I'm thinking we should move to cleartext client ids in puppet.

> ::: modules/balrog_scriptworker/manifests/settings.pp:38
> (Diff revision 1)
> >              balrog_api_root => 'https://aus4-admin.mozilla.org/api',
> > +
> > +            dummy => false,
> > +            tools_repo => 'https://hg.mozilla.org/build/tools',
> > +            taskcluster_client_id => 'project/releng/scriptworker/balrogworker',
> > +            taskcluster_access_token => secret('balrogworker_prod_taskcluster_access_token'),
> 
> Similar problem, dev vs prod credentials seem to match. I suppose this is
> expected since we didn't generate a new set but just thought it's worth
> raising.

Same here: the prod secrets are currently named `balrogworker_dev*` so I copied `balrogworker_dev*` to `balrogworker_prod*` but haven't replaced them in hiera yet. The new dev creds are in my env.

> ::: modules/balrog_scriptworker/templates/script_config.json.erb:11
> (Diff revision 1)
> >  
> >      "verbose": <%= scope.lookupvar("balrog_scriptworker::settings::verbose_logging") %>,
> > -    "dummy": false,
> > +    "dummy": @env_config["dummy"],
> >      "disable_certs": false,
> >  
> >      "server_config": {
> 
> Question: I suppose dev vs production will be equally safe from a security
> perspective right? I mean, there's no reason to ease the security on the dev
> balrogworkers as opposed to the other one. Otherwise, I'm thinking we should
> split these configs somehow the same way as you did for beetmover since prod
> passwords would get this way on the dev-balrogworkers instances.

For balrog, the server passwords + api roots are shifted for dev in settings.pp, so we never have creds for prod on the dev instance... we can ask to push to the nightly server, and it'll still push to dep. I agree this is possibly more dangerous than if we only supported the dep server for dev. This is on my mental list of things to possibly revisit.

Comment 10

[Aki Sasaki (not active)]

Attachment: beetmoverworker_client_id.diff

Follow suit with beetmoverworker.

Comment 11

[Mihai Tabara [:mtabara]⌚️GMT]

(In reply to Aki Sasaki [:aki] from comment #8)
> > ::: modules/beetmover_scriptworker/manifests/settings.pp:33
> > (Diff revision 1)
> > >              dep_beetmover_aws_s3_firefox_bucket     => 'net-mozaws-stage-delivery-firefox',
> > >              dep_beetmover_aws_s3_fennec_bucket      => 'net-mozaws-stage-delivery-archive',
> > > +
> > > +            config_template                         => 'beetmover_scriptworker/dev_script_config.json.erb',
> > > +            worker_type                             => 'beetmover-dev',
> > > +            taskcluster_client_id                   => secret('beetmoverworker_dev_taskcluster_client_id'),
> > 
> > They seem to match the production credentials. Is this expected or did we
> > not generate a separate set of credentials just yet?
> 
> This is because the current prod secrets are called `beetmover_dev*`. Rather
> than create a new secret named `beetmover_dev_for_realz*` I copied the
> `beetmover_dev*` secrets to `beetmover_prod*`. I haven't replaced the
> `beetmover_dev*` secrets because without this patch, that would replace the
> current beetmover scriptworkers' secrets with the new dev secrets.
> 
> The new generated secrets are in my environment; beetmover-dev1 is pinned to
> that env.

Got it, makes sense!
> 
> > ::: modules/beetmover_scriptworker/templates/base_script_config.json.erb:37
> > (Diff revision 1)
> > >              "fennec_candidates": "<%= scope.lookupvar("beetmover_scriptworker::settings::root") %>/lib/python3.5/site-packages/beetmoverscript/templates/fennec_candidates.yml",
> > >              "fennec_candidates_repacks": "<%= scope.lookupvar("beetmover_scriptworker::settings::root") %>/lib/python3.5/site-packages/beetmoverscript/templates/fennec_candidates_repacks.yml",
> > >              "fennecx86_candidates": "<%= scope.lookupvar("beetmover_scriptworker::settings::root") %>/lib/python3.5/site-packages/beetmoverscript/templates/fennecx86_candidates.yml",
> > >              "fennecaarch64_candidates": "<%= scope.lookupvar("beetmover_scriptworker::settings::root") %>/lib/python3.5/site-packages/beetmoverscript/templates/fennecaarch64_candidates.yml"
> > >          },
> > > -        "push-to-releases": {},
> > > +        "push-to-releases": {}
> > 
> > I think we can trim this as well until we add beetmoverscript
> > support(https://github.com/mozilla-releng/beetmoverscript/blob/master/
> > beetmoverscript/script.py#L80) to replace the existing mozharness candidates
> > => releases copyover
> > script(https://hg.mozilla.org/mozilla-central/file/tip/testing/mozharness/
> > scripts/release/push-candidate-to-releases.py)
> 
> I'm adding that now :)

You're too fast! :P

Comment 12

[Mihai Tabara [:mtabara]⌚️GMT]

(In reply to Aki Sasaki [:aki] from comment #9)
> Yes. I debated doing this -- if we use an env, we can switch the shared
> tools repo without worry. But we might want to pin dev against a tools repo
> in prod puppet for some reason; this would allow for that.

Makes sense, thanks for explaining!

> I thought about it.
> 
> Arguments for cleartext client ids:
> - IMO it's easier to slip up if everything's in hiera -- it's unclear what
> client id we're using, and we have to ssh in and decrypt it to figure it out.
>   - similarly, it's hard to know which client to update if we want to adjust
> scopes.
> - the client id isn't secret.
> - fewer hiera edits
> 
> Arguments against:
> - consistency
> - ???
> 
> I'm thinking we should move to cleartext client ids in puppet.

Okay, I think you're right. I remember it used to be like that before scriptworkers. The Ids for old relpro puppet stuff would be cleartext, or at least the vast majority. I'm fine going this way from now on and edit on the fly as we work in various code pieces.
 
> For balrog, the server passwords + api roots are shifted for dev in
> settings.pp, so we never have creds for prod on the dev instance... we can
> ask to push to the nightly server, and it'll still push to dep. I agree this
> is possibly more dangerous than if we only supported the dep server for dev.
> This is on my mental list of things to possibly revisit.

:+1

Comment 13

[Aki Sasaki (not active)]

Thanks! And you were right, there was one more beetmover-dev vs beetmoverworker-dev =\
Boo... fixed, though.

Comment 14

[Nick Thomas [:nthomas] (UTC+12)]

Followup fix for balrog bustage - https://hg.mozilla.org/build/puppet/rev/c5e97f753829. Thanks for being so responsive Aki.