1225899 - add beet mover tasks to releasetasks task graph

Status: RESOLVED FIXED

(Release Engineering :: Release Automation, defect)

Description

[Jordan Lund (:jlund)]

This bug tracks the scheduling of all things beet mover.

there will be a task for each

1) en_us build (the promoted CI builds)
2) l10n repack
3) partial

they will call a task that runs mozharness with something like:

python scripts/beet_mover.py --template configs/beetmover/locale.jinja.template --platform win32 --version 44.0 --locale en_us --taskid xMLQpY17T16xqdPpOm1uTg --generate-manifest --verify-bits --upload-bits

Comment 1

[Jordan Lund (:jlund)]

took a look at this today and yesterday. should have some patches up tomorrow am.

Comment 2

[Jordan Lund (:jlund)]

Attachment: 160117_bug_1225899_beetmover_en_US-releaserunner.patch

this is for part 1: en-US complete push to candidates

looks like our date copy of release-runner is not managed by puppet so I'll need to update release-runner.ini to reflect the aws creds addition

Comment 3

[Rail Aliiev [:rail]]

Comment on attachment 8708891 [details] [diff] [review]
160117_bug_1225899_beetmover_en_US-releaserunner.patch

Review of attachment 8708891 [details] [diff] [review]:
-----------------------------------------------------------------

::: buildfarm/release/release-runner.py
@@ +171,5 @@
>              platform=platform,
>          ))
>          platforms[platform] = {
> +            "task_id": task["taskId"],
> +            "upload_platform": upload_platform_mapping[platform],

Please use buildbot2ftp instead, see https://dxr.mozilla.org/build-central/source/tools/lib/python/release/platforms.py#75

@@ +229,5 @@
>      configs_workdir = 'buildbot-configs'
>      balrog_username = get_config(config, "balrog", "username", None)
>      balrog_password = get_config(config, "balrog", "password", None)
> +    beetmover_aws_access_key_id = get_config(config, "beetmover", "aws_access_key_id", None)
> +    beetmover_aws_secret_access_key = get_config(config, "beetmover", "aws_secret_access_key", None)

These are coming from puppet, correct?

@@ +302,5 @@
>                  "signing_class": "dep-signing",
>                  "bouncer_enabled": branchConfig["bouncer_enabled"],
>                  "release_channels": branchConfig["release_channels"],
>                  "signing_pvt_key": signing_pvt_key,
> +                "push_to_candidates_enabled": True

Hmm, sounds more like a branch config variable to me.

Comment 4

[Jordan Lund (:jlund)]

Attachment: 160118_bug_1225899_beetmover_en-US-tools.patch

thanks for review :)

interdiff: http://people.mozilla.org/~jlund/160118_bug_1225899_beetmover_en-US-tools-interdiff.patch

Comment 5

[Jordan Lund (:jlund)]

Attachment: 160118_bug_1225899_beetmover_en-US-bbot-cfgs.patch

Comment 6

[Jordan Lund (:jlund)]

Attachment: releasetasks beetmover en-US completes

updated to reflect review comments

Comment 7

[Rail Aliiev [:rail]]

Comment on attachment 8709157 [details] [diff] [review]
160118_bug_1225899_beetmover_en-US-tools.patch

Review of attachment 8709157 [details] [diff] [review]:
-----------------------------------------------------------------

::: buildfarm/release/release-runner.py
@@ +164,5 @@
>              platform=platform,
>          ))
>          platforms[platform] = {
> +            "task_id": task["taskId"],
> +            "upload_platform": buildbot2ftp[platform],

A nit, buildbot2ftp is a function, need to call it:

  buildbot2ftp(platform)

@@ +222,5 @@
>      configs_workdir = 'buildbot-configs'
>      balrog_username = get_config(config, "balrog", "username", None)
>      balrog_password = get_config(config, "balrog", "password", None)
> +    beetmover_aws_access_key_id = get_config(config, "beetmover", "aws_access_key_id", None)
> +    beetmover_aws_secret_access_key = get_config(config, "beetmover", "aws_secret_access_key", None)

Don't forget to submit a puppet patch to add these variables to the config

@@ +295,5 @@
>                  "signing_class": "dep-signing",
>                  "bouncer_enabled": branchConfig["bouncer_enabled"],
>                  "release_channels": branchConfig["release_channels"],
>                  "signing_pvt_key": signing_pvt_key,
> +                "push_to_candidates_enabled": branchConfig['push_to_candidates_enabled']

Your comment in IRC, that this variable can be used as a environment-specific variable (prod/dev) makes sense to me. Feel free to use either approach.

Comment 8

[Jordan Lund (:jlund)]

(In reply to Rail Aliiev [:rail] from comment #7)
> Comment on attachment 8709157 [details] [diff] [review]
> 160118_bug_1225899_beetmover_en-US-tools.patch
> 
> Review of attachment 8709157 [details] [diff] [review]:
> -----------------------------------------------------------------
> 
> ::: buildfarm/release/release-runner.py
> @@ +164,5 @@
> >              platform=platform,
> >          ))
> >          platforms[platform] = {
> > +            "task_id": task["taskId"],
> > +            "upload_platform": buildbot2ftp[platform],
> 
> A nit, buildbot2ftp is a function, need to call it:
> 
>   buildbot2ftp(platform)
> 

LOL

> @@ +222,5 @@
> >      configs_workdir = 'buildbot-configs'
> >      balrog_username = get_config(config, "balrog", "username", None)
> >      balrog_password = get_config(config, "balrog", "password", None)
> > +    beetmover_aws_access_key_id = get_config(config, "beetmover", "aws_access_key_id", None)
> > +    beetmover_aws_secret_access_key = get_config(config, "beetmover", "aws_secret_access_key", None)
> 
> Don't forget to submit a puppet patch to add these variables to the config

oh right. I thought because puppet doesn't manage our date instance that it wasn't puppetized yet. But I guess I can add it to https://dxr.mozilla.org/build-central/source/puppet/modules/releaserunner/templates/release-runner.ini.erb which will be shared by production releaserunner too and be a no-op for that.

> @@ +295,5 @@
> >                  "signing_class": "dep-signing",
> >                  "bouncer_enabled": branchConfig["bouncer_enabled"],
> >                  "release_channels": branchConfig["release_channels"],
> >                  "signing_pvt_key": signing_pvt_key,
> > +                "push_to_candidates_enabled": branchConfig['push_to_candidates_enabled']
> 
> Your comment in IRC, that this variable can be used as a
> environment-specific variable (prod/dev) makes sense to me. Feel free to use
> either approach.

I'll leave this as is. we will want 'push_to_releases_enabled' to be a branchconfig item so I can keep those side by side.

Comment 9

[Jordan Lund (:jlund)]

Attachment: 160118_bug_1225899_beetmover_en-US-puppet.patch

see motivation from: https://bugzilla.mozilla.org/show_bug.cgi?id=1225899#c8

Comment 10

[Jordan Lund (:jlund)]

Comment on attachment 8709206 [details] [diff] [review]
160118_bug_1225899_beetmover_en-US-puppet.patch

ftr - secrets are in secrets.eyaml

Comment 11

[Jordan Lund (:jlund)]

Comment on attachment 8709159 [details] [diff] [review]
160118_bug_1225899_beetmover_en-US-bbot-cfgs.patch

https://hg.mozilla.org/build/buildbot-configs/rev/fca43032761e

Comment 12

[Jordan Lund (:jlund)]

Attachment: 160118_bug_1225899_beetmover_en-US-tools-2.patch

carrying r+

interdiff:
diff --git a/buildfarm/release/release-runner.py b/buildfarm/release/release-runner.py
index c1cc4e1..1512a60 100755
--- a/buildfarm/release/release-runner.py
+++ b/buildfarm/release/release-runner.py
@@ -165,7 +165,7 @@ def get_en_US_config(release, branchConfig, branch, index):
         ))
         platforms[platform] = {
             "task_id": task["taskId"],
-            "upload_platform": buildbot2ftp[platform],
+            "upload_platform": buildbot2ftp(platform),
         }

     return {

Comment 13

[Release Engineering SlaveAPI Service]

In production: https://hg.mozilla.org/build/buildbot-configs/rev/fca43032761e

Comment 14

[Jordan Lund (:jlund)]

Comment on attachment 8709206 [details] [diff] [review]
160118_bug_1225899_beetmover_en-US-puppet.patch

thanks,

checked in and merged to prod

remote:   https://hg.mozilla.org/build/puppet/rev/3d9034df8272
remote:   https://hg.mozilla.org/build/puppet/rev/0928dc3fed91

Comment 15

[Jordan Lund (:jlund)]

Attachment: beetmover template fix ups

Comment 16

[Jordan Lund (:jlund)]

Attachment: 160120_bug_1225899_beetmover_en-US-remove_unneeded_buildbot2ftp-tools.patch

see https://github.com/mozilla/releasetasks/pull/86 for motivation

Comment 17

[Jordan Lund (:jlund)]

Comment on attachment 8710201 [details] [diff] [review]
160120_bug_1225899_beetmover_en-US-remove_unneeded_buildbot2ftp-tools.patch

ftr - I've landed this for now so I can test it.

Comment 18

[Jordan Lund (:jlund)]

Comment on attachment 8710200 [details] [review]
beetmover template fix ups

ftr - I've landed this for now so I can test it. still needs a stamp

Comment 19

[Jordan Lund (:jlund)]

hmm, release-runner failed to generate a usable task graph. 

full log is long because of the output of the task graph in json. end of log: http://people.mozilla.org/~jlund/failed_beetmover_release-runner.log

Odd that our tests didn't catch this. Will have to look more tomorrow.

Comment 20

[Jordan Lund (:jlund)]

Attachment: fix indent level for treeherder env so that it is nested under extra

I probably shouldn't be doing it this way. Maybe I can just move my own copy of releasetasks to /home/cltbld/releasetasks and hack on that for a bit till it works rather than pollute github:mozilla/releasetasks with small fixes at a time.

at any rate, this patch tries to tackle the issues I outlined in previous comment and it seems to get us past the issue. We even have a task graph submitted with beetmover tasks: https://tools.taskcluster.net/task-graph-inspector/#y0h2HaBZTl6yniUziS6vqA/

Comment 21

[Jordan Lund (:jlund)]

looking good now: https://console.aws.amazon.com/s3/home?region=us-east-1&bucket=mozilla-releng-beet-mover-dev&prefix=pub/firefox/candidates/46.0b39-candidates/build1/

next up, repacks :)

note, `encryptedEnv:` doesn't work unless you have at least one `env:` var defined as well :(

Comment 22

[Jordan Lund (:jlund)]

Attachment: adds repack l10n beetmover candidates tasks

Comment 23

[Rail Aliiev [:rail]]

Comment on attachment 8714140 [details] [review]
adds repack l10n beetmover candidates tasks

Commented in the PR. Please, fix the deadline/valid_for issue before you land.

Comment 24

[Jordan Lund (:jlund)]

Attachment: beetmover releasetasks partials

Comment 25

[Jordan Lund (:jlund)]

Comment on attachment 8716679 [details] [review]
beetmover releasetasks partials

associated mozharness patch: https://bugzilla.mozilla.org/attachment.cgi?id=8716677