1441353 - Create a scriptworker for submission to and fetching from AMO

Status: RESOLVED FIXED

(Release Engineering :: General, enhancement)

Description

[Justin Wood (:Callek)]

We need to submit addons to AMO for signing and get the signed xpi back for language packs.

Comment 1

[Johan Lorenzo [:jlorenzo]]

Attachment: Bug 1441353 - part 1: Add AMO tasks for signing and uploading

Review commit: https://reviewboard.mozilla.org/r/230698/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/230698/

Comment 2

[Johan Lorenzo [:jlorenzo]]

Comment on attachment 8961871 [details]
Bug 1441353 - part 1: Add AMO tasks for signing and uploading

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/230698/diff/1-2/

Comment 3

[Code Review Bot [:reviewbot]]

Comment on attachment 8961871 [details]
Bug 1441353 - part 1: Add AMO tasks for signing and uploading

https://reviewboard.mozilla.org/r/230698/#review237164


Code analysis found 14 defects in this patch:
 - 14 defects found by mozlint

You can run this analysis locally with:
 - `./mach lint path/to/file` (JS/Python)


If you see a problem in this automated review, please report it here: http://bit.ly/2y9N9Vx


::: taskcluster/ci/release-sign-and-push-langpacks/kind.yml:33
(Diff revision 2)
> +         mozilla-release: scriptworker-prov-v1/pushaddon-v1
> +         default: scriptworker-prov-v1/pushaddon-dev
> +   worker:
> +      implementation: sign-and-push-addons
> +      channel:
> +          by-project:

Error: Wrong indentation: expected 9 but found 10 [yamllint: indentation]

::: taskcluster/ci/release-sign-and-push-langpacks/kind.yml:37
(Diff revision 2)
> +      channel:
> +          by-project:
> +             # Only release langpacks are listed publicly
> +             mozilla-release: listed
> +             default: unlisted
> +      upstream-artifacts: # See transforms

Warning: Too few spaces before comment [yamllint: comments]

::: taskcluster/ci/release-sign-and-push-langpacks/kind.yml:47
(Diff revision 2)
> +            - project:releng:addons.mozilla.org:server:production
> +         mozilla-release:
> +            - project:releng:addons.mozilla.org:server:production
> +         default:
> +            - project:releng:addons.mozilla.org:server:staging
> +   run-on-projects: []

Error: Duplication of key "run-on-projects" in mapping [yamllint: key-duplicates]

::: taskcluster/taskgraph/transforms/release_sign_and_push_langpacks.py:10
(Diff revision 2)
> +Transform the beetmover task into an actual task description.
> +"""
> +
> +from __future__ import absolute_import, print_function, unicode_literals
> +
> +import re

Error: 're' imported but unused [flake8: F401]

::: taskcluster/taskgraph/transforms/release_sign_and_push_langpacks.py:14
(Diff revision 2)
> +
> +import re
> +
> +from taskgraph.transforms.base import TransformSequence
> +from taskgraph.util.attributes import copy_attributes_from_dependent_job
> +from taskgraph.util.schema import validate_schema, Schema, resolve_keyed_by

Error: 'taskgraph.util.schema.schema' imported but unused [flake8: F401]

::: taskcluster/taskgraph/transforms/release_sign_and_push_langpacks.py:14
(Diff revision 2)
> +
> +import re
> +
> +from taskgraph.transforms.base import TransformSequence
> +from taskgraph.util.attributes import copy_attributes_from_dependent_job
> +from taskgraph.util.schema import validate_schema, Schema, resolve_keyed_by

Error: 'taskgraph.util.schema.validate_schema' imported but unused [flake8: F401]

::: taskcluster/taskgraph/transforms/release_sign_and_push_langpacks.py:15
(Diff revision 2)
> +import re
> +
> +from taskgraph.transforms.base import TransformSequence
> +from taskgraph.util.attributes import copy_attributes_from_dependent_job
> +from taskgraph.util.schema import validate_schema, Schema, resolve_keyed_by
> +from taskgraph.util.scriptworker import (get_beetmover_bucket_scope,

Error: 'taskgraph.util.scriptworker.get_beetmover_action_scope' imported but unused [flake8: F401]

::: taskcluster/taskgraph/transforms/release_sign_and_push_langpacks.py:15
(Diff revision 2)
> +import re
> +
> +from taskgraph.transforms.base import TransformSequence
> +from taskgraph.util.attributes import copy_attributes_from_dependent_job
> +from taskgraph.util.schema import validate_schema, Schema, resolve_keyed_by
> +from taskgraph.util.scriptworker import (get_beetmover_bucket_scope,

Error: 'taskgraph.util.scriptworker.get_phase' imported but unused [flake8: F401]

::: taskcluster/taskgraph/transforms/release_sign_and_push_langpacks.py:15
(Diff revision 2)
> +import re
> +
> +from taskgraph.transforms.base import TransformSequence
> +from taskgraph.util.attributes import copy_attributes_from_dependent_job
> +from taskgraph.util.schema import validate_schema, Schema, resolve_keyed_by
> +from taskgraph.util.scriptworker import (get_beetmover_bucket_scope,

Error: 'taskgraph.util.scriptworker.get_beetmover_bucket_scope' imported but unused [flake8: F401]

::: taskcluster/taskgraph/transforms/release_sign_and_push_langpacks.py:18
(Diff revision 2)
> +from taskgraph.util.attributes import copy_attributes_from_dependent_job
> +from taskgraph.util.schema import validate_schema, Schema, resolve_keyed_by
> +from taskgraph.util.scriptworker import (get_beetmover_bucket_scope,
> +                                         get_beetmover_action_scope,
> +                                         get_phase)
> +from taskgraph.transforms.task import task_description_schema

Error: 'taskgraph.transforms.task.task_description_schema' imported but unused [flake8: F401]

::: taskcluster/taskgraph/transforms/release_sign_and_push_langpacks.py:19
(Diff revision 2)
> +from taskgraph.util.schema import validate_schema, Schema, resolve_keyed_by
> +from taskgraph.util.scriptworker import (get_beetmover_bucket_scope,
> +                                         get_beetmover_action_scope,
> +                                         get_phase)
> +from taskgraph.transforms.task import task_description_schema
> +from voluptuous import Any, Required, Optional

Error: 'voluptuous.optional' imported but unused [flake8: F401]

::: taskcluster/taskgraph/transforms/release_sign_and_push_langpacks.py:19
(Diff revision 2)
> +from taskgraph.util.schema import validate_schema, Schema, resolve_keyed_by
> +from taskgraph.util.scriptworker import (get_beetmover_bucket_scope,
> +                                         get_beetmover_action_scope,
> +                                         get_phase)
> +from taskgraph.transforms.task import task_description_schema
> +from voluptuous import Any, Required, Optional

Error: 'voluptuous.required' imported but unused [flake8: F401]

::: taskcluster/taskgraph/transforms/release_sign_and_push_langpacks.py:19
(Diff revision 2)
> +from taskgraph.util.schema import validate_schema, Schema, resolve_keyed_by
> +from taskgraph.util.scriptworker import (get_beetmover_bucket_scope,
> +                                         get_beetmover_action_scope,
> +                                         get_phase)
> +from taskgraph.transforms.task import task_description_schema
> +from voluptuous import Any, Required, Optional

Error: 'voluptuous.any' imported but unused [flake8: F401]

::: taskcluster/taskgraph/transforms/release_sign_and_push_langpacks.py:63
(Diff revision 2)
> +
> +        treeherder = job.get('treeherder', {})
> +        treeherder.setdefault('symbol', 'langpack(SnP{})'.format(attributes.get('l10n_chunk', '')))
> +        dep_th_platform = dep_job.task.get('extra', {}).get(
> +            'treeherder', {}).get('machine', {}).get('platform', ''
> +        )

Error: Closing bracket does not match visual indentation [flake8: E124]

Comment 4

[Johan Lorenzo [:jlorenzo]]

Comment on attachment 8961871 [details]
Bug 1441353 - part 1: Add AMO tasks for signing and uploading

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/230698/diff/2-3/

Comment 5

[Code Review Bot [:reviewbot]]

Comment on attachment 8961871 [details]
Bug 1441353 - part 1: Add AMO tasks for signing and uploading

https://reviewboard.mozilla.org/r/230698/#review237170


Code analysis found 14 defects in this patch:
 - 14 defects found by mozlint

You can run this analysis locally with:
 - `./mach lint path/to/file` (JS/Python)


If you see a problem in this automated review, please report it here: http://bit.ly/2y9N9Vx


::: taskcluster/ci/release-sign-and-push-langpacks/kind.yml:33
(Diff revision 3)
> +         mozilla-release: scriptworker-prov-v1/pushaddon-v1
> +         default: scriptworker-prov-v1/pushaddon-dev
> +   worker:
> +      implementation: sign-and-push-addons
> +      channel:
> +          by-project:

Error: Wrong indentation: expected 9 but found 10 [yamllint: indentation]

::: taskcluster/ci/release-sign-and-push-langpacks/kind.yml:37
(Diff revision 3)
> +      channel:
> +          by-project:
> +             # Only release langpacks are listed publicly
> +             mozilla-release: listed
> +             default: unlisted
> +      upstream-artifacts: # See transforms

Warning: Too few spaces before comment [yamllint: comments]

::: taskcluster/ci/release-sign-and-push-langpacks/kind.yml:47
(Diff revision 3)
> +            - project:releng:addons.mozilla.org:server:production
> +         mozilla-release:
> +            - project:releng:addons.mozilla.org:server:production
> +         default:
> +            - project:releng:addons.mozilla.org:server:staging
> +   run-on-projects: []

Error: Duplication of key "run-on-projects" in mapping [yamllint: key-duplicates]

::: taskcluster/taskgraph/transforms/release_sign_and_push_langpacks.py:10
(Diff revision 3)
> +Transform the beetmover task into an actual task description.
> +"""
> +
> +from __future__ import absolute_import, print_function, unicode_literals
> +
> +import re

Error: 're' imported but unused [flake8: F401]

::: taskcluster/taskgraph/transforms/release_sign_and_push_langpacks.py:14
(Diff revision 3)
> +
> +import re
> +
> +from taskgraph.transforms.base import TransformSequence
> +from taskgraph.util.attributes import copy_attributes_from_dependent_job
> +from taskgraph.util.schema import validate_schema, Schema, resolve_keyed_by

Error: 'taskgraph.util.schema.schema' imported but unused [flake8: F401]

::: taskcluster/taskgraph/transforms/release_sign_and_push_langpacks.py:14
(Diff revision 3)
> +
> +import re
> +
> +from taskgraph.transforms.base import TransformSequence
> +from taskgraph.util.attributes import copy_attributes_from_dependent_job
> +from taskgraph.util.schema import validate_schema, Schema, resolve_keyed_by

Error: 'taskgraph.util.schema.validate_schema' imported but unused [flake8: F401]

::: taskcluster/taskgraph/transforms/release_sign_and_push_langpacks.py:15
(Diff revision 3)
> +import re
> +
> +from taskgraph.transforms.base import TransformSequence
> +from taskgraph.util.attributes import copy_attributes_from_dependent_job
> +from taskgraph.util.schema import validate_schema, Schema, resolve_keyed_by
> +from taskgraph.util.scriptworker import (get_beetmover_bucket_scope,

Error: 'taskgraph.util.scriptworker.get_beetmover_action_scope' imported but unused [flake8: F401]

::: taskcluster/taskgraph/transforms/release_sign_and_push_langpacks.py:15
(Diff revision 3)
> +import re
> +
> +from taskgraph.transforms.base import TransformSequence
> +from taskgraph.util.attributes import copy_attributes_from_dependent_job
> +from taskgraph.util.schema import validate_schema, Schema, resolve_keyed_by
> +from taskgraph.util.scriptworker import (get_beetmover_bucket_scope,

Error: 'taskgraph.util.scriptworker.get_phase' imported but unused [flake8: F401]

::: taskcluster/taskgraph/transforms/release_sign_and_push_langpacks.py:15
(Diff revision 3)
> +import re
> +
> +from taskgraph.transforms.base import TransformSequence
> +from taskgraph.util.attributes import copy_attributes_from_dependent_job
> +from taskgraph.util.schema import validate_schema, Schema, resolve_keyed_by
> +from taskgraph.util.scriptworker import (get_beetmover_bucket_scope,

Error: 'taskgraph.util.scriptworker.get_beetmover_bucket_scope' imported but unused [flake8: F401]

::: taskcluster/taskgraph/transforms/release_sign_and_push_langpacks.py:18
(Diff revision 3)
> +from taskgraph.util.attributes import copy_attributes_from_dependent_job
> +from taskgraph.util.schema import validate_schema, Schema, resolve_keyed_by
> +from taskgraph.util.scriptworker import (get_beetmover_bucket_scope,
> +                                         get_beetmover_action_scope,
> +                                         get_phase)
> +from taskgraph.transforms.task import task_description_schema

Error: 'taskgraph.transforms.task.task_description_schema' imported but unused [flake8: F401]

::: taskcluster/taskgraph/transforms/release_sign_and_push_langpacks.py:19
(Diff revision 3)
> +from taskgraph.util.schema import validate_schema, Schema, resolve_keyed_by
> +from taskgraph.util.scriptworker import (get_beetmover_bucket_scope,
> +                                         get_beetmover_action_scope,
> +                                         get_phase)
> +from taskgraph.transforms.task import task_description_schema
> +from voluptuous import Any, Required, Optional

Error: 'voluptuous.optional' imported but unused [flake8: F401]

::: taskcluster/taskgraph/transforms/release_sign_and_push_langpacks.py:19
(Diff revision 3)
> +from taskgraph.util.schema import validate_schema, Schema, resolve_keyed_by
> +from taskgraph.util.scriptworker import (get_beetmover_bucket_scope,
> +                                         get_beetmover_action_scope,
> +                                         get_phase)
> +from taskgraph.transforms.task import task_description_schema
> +from voluptuous import Any, Required, Optional

Error: 'voluptuous.required' imported but unused [flake8: F401]

::: taskcluster/taskgraph/transforms/release_sign_and_push_langpacks.py:19
(Diff revision 3)
> +from taskgraph.util.schema import validate_schema, Schema, resolve_keyed_by
> +from taskgraph.util.scriptworker import (get_beetmover_bucket_scope,
> +                                         get_beetmover_action_scope,
> +                                         get_phase)
> +from taskgraph.transforms.task import task_description_schema
> +from voluptuous import Any, Required, Optional

Error: 'voluptuous.any' imported but unused [flake8: F401]

::: taskcluster/taskgraph/transforms/release_sign_and_push_langpacks.py:63
(Diff revision 3)
> +
> +        treeherder = job.get('treeherder', {})
> +        treeherder.setdefault('symbol', 'langpack(SnP{})'.format(attributes.get('l10n_chunk', '')))
> +        dep_th_platform = dep_job.task.get('extra', {}).get(
> +            'treeherder', {}).get('machine', {}).get('platform', ''
> +        )

Error: Closing bracket does not match visual indentation [flake8: E124]

Comment 6

[Johan Lorenzo [:jlorenzo]]

Comment on attachment 8961871 [details]
Bug 1441353 - part 1: Add AMO tasks for signing and uploading

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/230698/diff/3-4/

Comment 7

[Code Review Bot [:reviewbot]]

Comment on attachment 8961871 [details]
Bug 1441353 - part 1: Add AMO tasks for signing and uploading

https://reviewboard.mozilla.org/r/230698/#review237196


Code analysis found 15 defects in this patch:
 - 15 defects found by mozlint

You can run this analysis locally with:
 - `./mach lint path/to/file` (JS/Python)


If you see a problem in this automated review, please report it here: http://bit.ly/2y9N9Vx


::: taskcluster/ci/release-sign-and-push-langpacks/kind.yml:33
(Diff revision 4)
> +         mozilla-release: scriptworker-prov-v1/pushaddon-v1
> +         default: scriptworker-prov-v1/pushaddon-dev
> +   worker:
> +      implementation: sign-and-push-addons
> +      channel:
> +          by-project:

Error: Wrong indentation: expected 9 but found 10 [yamllint: indentation]

::: taskcluster/ci/release-sign-and-push-langpacks/kind.yml:37
(Diff revision 4)
> +      channel:
> +          by-project:
> +             # Only release langpacks are listed publicly
> +             mozilla-release: listed
> +             default: unlisted
> +      upstream-artifacts: # See transforms

Warning: Too few spaces before comment [yamllint: comments]

::: taskcluster/ci/release-sign-and-push-langpacks/kind.yml:47
(Diff revision 4)
> +            - project:releng:addons.mozilla.org:server:production
> +         mozilla-release:
> +            - project:releng:addons.mozilla.org:server:production
> +         default:
> +            - project:releng:addons.mozilla.org:server:staging
> +   run-on-projects: []

Error: Duplication of key "run-on-projects" in mapping [yamllint: key-duplicates]

::: taskcluster/taskgraph/transforms/release_sign_and_push_langpacks.py:10
(Diff revision 4)
> +Transform the beetmover task into an actual task description.
> +"""
> +
> +from __future__ import absolute_import, print_function, unicode_literals
> +
> +import re

Error: 're' imported but unused [flake8: F401]

::: taskcluster/taskgraph/transforms/release_sign_and_push_langpacks.py:14
(Diff revision 4)
> +
> +import re
> +
> +from taskgraph.transforms.base import TransformSequence
> +from taskgraph.util.attributes import copy_attributes_from_dependent_job
> +from taskgraph.util.schema import validate_schema, Schema, resolve_keyed_by

Error: 'taskgraph.util.schema.schema' imported but unused [flake8: F401]

::: taskcluster/taskgraph/transforms/release_sign_and_push_langpacks.py:14
(Diff revision 4)
> +
> +import re
> +
> +from taskgraph.transforms.base import TransformSequence
> +from taskgraph.util.attributes import copy_attributes_from_dependent_job
> +from taskgraph.util.schema import validate_schema, Schema, resolve_keyed_by

Error: 'taskgraph.util.schema.validate_schema' imported but unused [flake8: F401]

::: taskcluster/taskgraph/transforms/release_sign_and_push_langpacks.py:15
(Diff revision 4)
> +import re
> +
> +from taskgraph.transforms.base import TransformSequence
> +from taskgraph.util.attributes import copy_attributes_from_dependent_job
> +from taskgraph.util.schema import validate_schema, Schema, resolve_keyed_by
> +from taskgraph.util.scriptworker import (get_beetmover_bucket_scope,

Error: 'taskgraph.util.scriptworker.get_beetmover_action_scope' imported but unused [flake8: F401]

::: taskcluster/taskgraph/transforms/release_sign_and_push_langpacks.py:15
(Diff revision 4)
> +import re
> +
> +from taskgraph.transforms.base import TransformSequence
> +from taskgraph.util.attributes import copy_attributes_from_dependent_job
> +from taskgraph.util.schema import validate_schema, Schema, resolve_keyed_by
> +from taskgraph.util.scriptworker import (get_beetmover_bucket_scope,

Error: 'taskgraph.util.scriptworker.get_phase' imported but unused [flake8: F401]

::: taskcluster/taskgraph/transforms/release_sign_and_push_langpacks.py:15
(Diff revision 4)
> +import re
> +
> +from taskgraph.transforms.base import TransformSequence
> +from taskgraph.util.attributes import copy_attributes_from_dependent_job
> +from taskgraph.util.schema import validate_schema, Schema, resolve_keyed_by
> +from taskgraph.util.scriptworker import (get_beetmover_bucket_scope,

Error: 'taskgraph.util.scriptworker.get_beetmover_bucket_scope' imported but unused [flake8: F401]

::: taskcluster/taskgraph/transforms/release_sign_and_push_langpacks.py:18
(Diff revision 4)
> +from taskgraph.util.attributes import copy_attributes_from_dependent_job
> +from taskgraph.util.schema import validate_schema, Schema, resolve_keyed_by
> +from taskgraph.util.scriptworker import (get_beetmover_bucket_scope,
> +                                         get_beetmover_action_scope,
> +                                         get_phase)
> +from taskgraph.transforms.task import task_description_schema

Error: 'taskgraph.transforms.task.task_description_schema' imported but unused [flake8: F401]

::: taskcluster/taskgraph/transforms/release_sign_and_push_langpacks.py:19
(Diff revision 4)
> +from taskgraph.util.schema import validate_schema, Schema, resolve_keyed_by
> +from taskgraph.util.scriptworker import (get_beetmover_bucket_scope,
> +                                         get_beetmover_action_scope,
> +                                         get_phase)
> +from taskgraph.transforms.task import task_description_schema
> +from voluptuous import Any, Required, Optional

Error: 'voluptuous.optional' imported but unused [flake8: F401]

::: taskcluster/taskgraph/transforms/release_sign_and_push_langpacks.py:19
(Diff revision 4)
> +from taskgraph.util.schema import validate_schema, Schema, resolve_keyed_by
> +from taskgraph.util.scriptworker import (get_beetmover_bucket_scope,
> +                                         get_beetmover_action_scope,
> +                                         get_phase)
> +from taskgraph.transforms.task import task_description_schema
> +from voluptuous import Any, Required, Optional

Error: 'voluptuous.required' imported but unused [flake8: F401]

::: taskcluster/taskgraph/transforms/release_sign_and_push_langpacks.py:19
(Diff revision 4)
> +from taskgraph.util.schema import validate_schema, Schema, resolve_keyed_by
> +from taskgraph.util.scriptworker import (get_beetmover_bucket_scope,
> +                                         get_beetmover_action_scope,
> +                                         get_phase)
> +from taskgraph.transforms.task import task_description_schema
> +from voluptuous import Any, Required, Optional

Error: 'voluptuous.any' imported but unused [flake8: F401]

::: taskcluster/taskgraph/transforms/release_sign_and_push_langpacks.py:44
(Diff revision 4)
> +        dep_job = job['dependent-task']
> +        dep_job_attributes = dep_job.attributes
> +        build_platform = dep_job_attributes.get('build_platform')
> +        if build_platform == 'linux64-nightly':
> +            yield job
> +        elif build_platform =='macosx64-nightly' and \

Error: Missing whitespace around operator [flake8: E225]

::: taskcluster/taskgraph/transforms/release_sign_and_push_langpacks.py:77
(Diff revision 4)
> +
> +        treeherder = job.get('treeherder', {})
> +        treeherder.setdefault('symbol', 'langpack(SnP{})'.format(attributes.get('l10n_chunk', '')))
> +        dep_th_platform = dep_job.task.get('extra', {}).get(
> +            'treeherder', {}).get('machine', {}).get('platform', ''
> +        )

Error: Closing bracket does not match visual indentation [flake8: E124]

Comment 8

[Johan Lorenzo [:jlorenzo]]

Attachment: Bug 1441353 - part 2: Add beetmover job to publish signed langpacks

Review commit: https://reviewboard.mozilla.org/r/232798/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/232798/

Comment 9

[Johan Lorenzo [:jlorenzo]]

Comment on attachment 8961871 [details]
Bug 1441353 - part 1: Add AMO tasks for signing and uploading

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/230698/diff/4-5/

Comment 10

[Johan Lorenzo [:jlorenzo]]

I added to [1]:
* queue:create-task:low:scriptworker-prov-v1/addon-dev
* project:releng:amo:staging

I also added to [2]:
* queue:create-task:highest:scriptworker-prov-v1/addon-v1
* project:releng:amo:production

[1] https://tools.taskcluster.net/auth/roles/moz-tree%3Alevel%3A1%3Agecko
[2] https://tools.taskcluster.net/auth/roles/moz-tree%3Alevel%3A3%3Agecko

Comment 11

[Johan Lorenzo [:jlorenzo]]

Attachment: Bug 1441353 - Add addon_scriptworker instances

Review commit: https://reviewboard.mozilla.org/r/233660/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/233660/

Comment 12

[Johan Lorenzo [:jlorenzo]]

Attachment: [cloud-tools] PR: Add addonworker instance type

Was r+'d by mtabara at https://github.com/mozilla-releng/build-cloud-tools/pull/334#pullrequestreview-109299594
Was merged on master at https://github.com/mozilla-releng/build-cloud-tools/commit/c36f06a07531fe56dcc87c9811c1cb66a8d9df79

Comment 13

[Johan Lorenzo [:jlorenzo]]

Comment on attachment 8964918 [details]
Bug 1441353 - Add addon_scriptworker instances

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/233660/diff/1-2/

Comment 14

[Justin Wood (:Callek)]

Comment on attachment 8964918 [details]
Bug 1441353 - Add addon_scriptworker instances

https://reviewboard.mozilla.org/r/233660/#review240270

::: manifests/moco-nodes.pp
(Diff revision 2)
>  # Loaner for dividehex; bug 1445842 and 1447766
>  node 'ds-test1.srv.releng.mdc2.mozilla.com' {
>      $aspects = [ 'low-security' ]
>      include toplevel::server
>  }
> -

EOF change?

::: modules/addon_scriptworker/manifests/init.pp:94
(Diff revision 2)
> +
> +    $config_content = $addon_scriptworker::settings::script_config_content
> +    file {
> +        $addon_scriptworker::settings::script_config:
> +            require => Python35::Virtualenv[$addon_scriptworker::settings::root],
> +            content => inline_template("<%- require 'json' -%><%= JSON.pretty_generate(@config_content) %>");

TIL

Comment 15

[Justin Wood (:Callek)]

Comment on attachment 8964918 [details]
Bug 1441353 - Add addon_scriptworker instances

https://reviewboard.mozilla.org/r/233660/#review240272

I should note, I mostly skimmed the code here.

Comment 16

[Johan Lorenzo [:jlorenzo]]

Comment on attachment 8961871 [details]
Bug 1441353 - part 1: Add AMO tasks for signing and uploading

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/230698/diff/5-6/

Comment 17

[Johan Lorenzo [:jlorenzo]]

Comment on attachment 8963979 [details]
Bug 1441353 - part 2: Add beetmover job to publish signed langpacks

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/232798/diff/1-2/

Comment 18

[Johan Lorenzo [:jlorenzo]]

Comment on attachment 8961871 [details]
Bug 1441353 - part 1: Add AMO tasks for signing and uploading

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/230698/diff/6-7/

Comment 19

[Johan Lorenzo [:jlorenzo]]

Comment on attachment 8963979 [details]
Bug 1441353 - part 2: Add beetmover job to publish signed langpacks

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/232798/diff/2-3/

Comment 20

[Code Review Bot [:reviewbot]]

Comment on attachment 8963979 [details]
Bug 1441353 - part 2: Add beetmover job to publish signed langpacks

https://reviewboard.mozilla.org/r/232798/#review240912


Code analysis found 3 defects in this patch:
 - 3 defects found by mozlint

You can run this analysis locally with:
 - `./mach lint path/to/file` (JS/Python)


If you see a problem in this automated review, please report it here: http://bit.ly/2y9N9Vx


::: taskcluster/taskgraph/transforms/release_beetmover_signed_addons.py:86
(Diff revision 3)
> +        resolve_keyed_by(
> +            job, 'worker-type', item_name=job['label'], project=config.params['project']
> +        )
> +        yield job
> +
> [e-mail]

Error: Expected 2 blank lines, found 1 [flake8: E302]

::: taskcluster/taskgraph/transforms/release_beetmover_signed_addons.py:110
(Diff revision 3)
> +            locales='/'.join(job['attributes']['chunk_locales']),
> +            platform=job['attributes']['build_platform']
> +        )
> +
> +
> +        job['scopes'] = [

Error: Too many blank lines (2) [flake8: E303]

::: taskcluster/taskgraph/transforms/release_beetmover_signed_addons.py:176
(Diff revision 3)
> +            platform_job = copy.deepcopy(job)
> +
> +            platform_job['attributes']['build_platform'] = platform
> +            platform_job['label'] = job['label'].replace('linux64', platform)
> +            platform_job['description'] = job['description'].replace('linux64', platform)
> +            platform_job['treeherder']['platform'] = platform_job['treeherder']['platform'].replace('linux64', platform)

Error: Line too long (120 > 99 characters) [flake8: E501]

Comment 21

[Justin Wood (:Callek)]

Comment on attachment 8961871 [details]
Bug 1441353 - part 1: Add AMO tasks for signing and uploading

https://reviewboard.mozilla.org/r/230698/#review241012

::: taskcluster/taskgraph/transforms/release_sign_and_push_langpacks.py:5
(Diff revision 7)
> +# This Source Code Form is subject to the terms of the Mozilla Public
> +# License, v. 2.0. If a copy of the MPL was not distributed with this
> +# file, You can obtain one at http://mozilla.org/MPL/2.0/.
> +"""
> +Transform the beetmover task into an actual task description.

nit: docstring message

::: taskcluster/taskgraph/transforms/release_sign_and_push_langpacks.py:49
(Diff revision 7)
> +def set_label(config, jobs):
> +    for job in jobs:
> +        label = 'sign-and-push-langpacks-{}'.format(job['dependent-task'].label)
> +        job['label'] = label
> +
> +        yield job

I wonder if the name_sanity transform could help here (not a blocker)

::: taskcluster/taskgraph/transforms/release_sign_and_push_langpacks.py:73
(Diff revision 7)
> +        if build_platform == 'linux64-nightly':
> +            yield job
> +        elif build_platform == 'macosx64-nightly' and \
> +                'ja-JP-mac' in dep_job_attributes.get('chunk_locales', ['en-US']):
> +            # Other locales of the same job shouldn't be processed
> +            dep_job_attributes['chunk_locales'] = ['ja-JP-mac']

I didn't confirm yet, but I'd want to be sure this doesn't mutate an object on the osx l10n job itself (as in, we don't end up stripping out other locales from signing/beetmover/etc for normal-l10n, possibly ordering-dependent)

::: taskcluster/taskgraph/transforms/release_sign_and_push_langpacks.py:104
(Diff revision 7)
> +        treeherder = job.get('treeherder', {})
> +        treeherder.setdefault('symbol', 'langpack(SnP{})'.format(attributes.get('l10n_chunk', '')))
> +        dep_th_platform = dep_job.task.get('extra', {}).get(
> +            'treeherder', {}).get('machine', {}).get('platform', '')
> +        treeherder.setdefault('platform', '{}/opt'.format(dep_th_platform))
> +        treeherder.setdefault('tier', 2)

Since this is release blocking, tier 1?

::: taskcluster/taskgraph/transforms/release_sign_and_push_langpacks.py:111
(Diff revision 7)
> +
> +        dependent_kind = str(dep_job.kind)
> +        dependencies = {dependent_kind: dep_job.label}
> +
> +        job['attributes'] = copy_attributes_from_dependent_job(dep_job)
> +        job['attributes']['chunk_locales'] = dep_job.attributes.get('chunk_locales', ['en-US'])

This is where we might want to strip OSX locales down to just ja-JP-mac (on the one matching task), to be safe.

Comment 22

[Johan Lorenzo [:jlorenzo]]

Comment on attachment 8961871 [details]
Bug 1441353 - part 1: Add AMO tasks for signing and uploading

https://reviewboard.mozilla.org/r/230698/#review241012

> I wonder if the name_sanity transform could help here (not a blocker)

Could be. I'm doing nasty stuff with label. That would require more clean up. I'd prefer to handle it in a follow up.

> This is where we might want to strip OSX locales down to just ja-JP-mac (on the one matching task), to be safe.

Great call. This was hiding a bug that I didn't catch before. We didn't have ja-JP-mac uploaded. This is now fixed.

Comment 23

[Johan Lorenzo [:jlorenzo]]

Comment on attachment 8961871 [details]
Bug 1441353 - part 1: Add AMO tasks for signing and uploading

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/230698/diff/7-8/

Comment 24

[Johan Lorenzo [:jlorenzo]]

Comment on attachment 8963979 [details]
Bug 1441353 - part 2: Add beetmover job to publish signed langpacks

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/232798/diff/3-4/

Comment 25

[Justin Wood (:Callek)]

Comment on attachment 8963979 [details]
Bug 1441353 - part 2: Add beetmover job to publish signed langpacks

https://reviewboard.mozilla.org/r/232798/#review241018

Please also take care of reviewbot's style concerns (except line length) and we're good.

::: taskcluster/taskgraph/transforms/beetmover_repackage.py
(Diff revision 3)
>      "target.web-platform.tests.tar.gz",
>      "target.xpcshell.tests.zip",
>      "target_info.txt",
>      "target.jsshell.zip",
>      "mozharness.zip",
> -    "target.langpack.xpi",

devedition likely still needs the en-US .xpi uploaded.

::: taskcluster/taskgraph/transforms/beetmover_repackage.py:75
(Diff revision 3)
>      r'^win(32|64)(|-devedition)-nightly$':
>          _DESKTOP_UPSTREAM_ARTIFACTS_UNSIGNED_EN_US + [
>              "host/bin/mar.exe",
>              "host/bin/mbsdiff.exe",
>          ],
> -    r'^(linux(|64)|macosx64|win(32|64))(|-devedition)-nightly-l10n$':
> +    r'^(linux(|64)|macosx64|win(32|64))(|-devedition)-nightly-l10n$': [],

Hrm, this breaks devedition, right? (we still want to publish langpacks for deved, just not yet signed)

::: taskcluster/taskgraph/transforms/release_beetmover_signed_addons.py:185
(Diff revisions 3 - 4)
> +
> +                yield platform_job
> +
>  
> +def _strip_ja_data_from_linux_job(platform_job):
> +    platform_job['description'] = platform_job['description'].replace('ja/', '')

This could use a comment explaining why we do this... (non blocking)

::: taskcluster/taskgraph/transforms/release_sign_and_push_langpacks.py:104
(Diff revision 3)
>          treeherder = job.get('treeherder', {})
>          treeherder.setdefault('symbol', 'langpack(SnP{})'.format(attributes.get('l10n_chunk', '')))
>          dep_th_platform = dep_job.task.get('extra', {}).get(
>              'treeherder', {}).get('machine', {}).get('platform', '')
>          treeherder.setdefault('platform', '{}/opt'.format(dep_th_platform))
> -        treeherder.setdefault('tier', 2)
> +        treeherder.setdefault('tier', 1)

huh how is this switching back to tier 2?

::: taskcluster/taskgraph/transforms/release_sign_and_push_langpacks.py:114
(Diff revision 3)
>  
>          job['description'] = job['description'].format(
>              locales='/'.join(job['attributes']['chunk_locales']),
>          )
>  
> -        job['dependencies'] = dependencies
> +        job['dependencies'] = {

in fact this file looks funny...

::: taskcluster/taskgraph/transforms/task.py:464
(Diff revision 3)
> -        Required('max-run-time'): int,
> +        Required('max-run-time', default=600): int,
>  
>          # locale key, if this is a locale beetmover job
>          Optional('locale'): basestring,
>  
> -        Required('release-properties'): {
> +        Optional('release-properties'): {

This is the one change in this patch I don't claim to know anything about.

Comment 26

[Justin Wood (:Callek)]

Comment on attachment 8963979 [details]
Bug 1441353 - part 2: Add beetmover job to publish signed langpacks

https://reviewboard.mozilla.org/r/232798/#review241048

::: taskcluster/taskgraph/transforms/beetmover_repackage.py:75
(Diff revision 4)
>      r'^win(32|64)(|-devedition)-nightly$':
>          _DESKTOP_UPSTREAM_ARTIFACTS_UNSIGNED_EN_US + [
>              "host/bin/mar.exe",
>              "host/bin/mbsdiff.exe",
>          ],
> -    r'^(linux(|64)|macosx64|win(32|64))(|-devedition)-nightly-l10n$':
> +    r'^(linux(|64)|macosx64|win(32|64))(|-devedition)-nightly-l10n$': [],

still have the devedition concern here...

Comment 27

[Johan Lorenzo [:jlorenzo]]

Comment on attachment 8963979 [details]
Bug 1441353 - part 2: Add beetmover job to publish signed langpacks

https://reviewboard.mozilla.org/r/232798/#review241018

> This is the one change in this patch I don't claim to know anything about.

That was an artifact of an old try of mine. Thank you for calling it out.

Comment 28

[Johan Lorenzo [:jlorenzo]]

Comment on attachment 8963979 [details]
Bug 1441353 - part 2: Add beetmover job to publish signed langpacks

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/232798/diff/4-5/

Comment 29

[Johan Lorenzo [:jlorenzo]]

Comment on attachment 8964918 [details]
Bug 1441353 - Add addon_scriptworker instances

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/233660/diff/2-3/

Comment 30

[Johan Lorenzo [:jlorenzo]]

Comment on attachment 8964918 [details]
Bug 1441353 - Add addon_scriptworker instances

https://reviewboard.mozilla.org/r/233660/#review240270

> EOF change?

Ah crap, I landed the patch whi the EOF change. I'll fix it next time.

Comment 31

[Johan Lorenzo [:jlorenzo]]

Comment on attachment 8964918 [details]
Bug 1441353 - Add addon_scriptworker instances

default: https://hg.mozilla.org/build/puppet/rev/a6db788c22109b564fcb0f5c9c897a2177c88210
production: https://hg.mozilla.org/build/puppet/rev/29d4bdb13bb2c491614c5a5c5d41f4ba82b7d952

Comment 32

[Johan Lorenzo [:jlorenzo]]

addonworker-dev-1 and addonworker-1 are now both attached to the production environment.

Comment 33

[Justin Wood (:Callek)]

Comment on attachment 8961871 [details]
Bug 1441353 - part 1: Add AMO tasks for signing and uploading

Approval Request Comment
[Feature/Bug causing the regression]: Language pack signing
[User impact if declined]: Language packs won't be signed and unable to enable signing required for langpacks
[Is this code covered by automated tests?]: No
[Has the fix been verified in Nightly?]: N/A
[Needs manual test from QE? If yes, steps to reproduce]:  No
[List of other uplifts needed for the feature/fix]: For full support product will need to enable the Langpack Signing pref and get QE testing, but this code enables releng to upload signed langpacks along with our automation.
[Is the change risky?]: Yes
[Why is the change risky/not risky?]: It affects the release process late in the cycle.
[String changes made/needed]: None

Mitigations to release risk:
* If there are failures relating to configuration on specific addons on AMO, and if releng can self-serve fix them, we will do so and rerun affected tasks.
* If there is a minor failure in the addonscript we'll deploy an updated version and rerun the affect task(s)
* If we're unable to accomplish the above within an hour, or if there is in-tree taskgraph bustage from this patch, we back out and instead target a beta next week.

Comment 34

[Justin Wood (:Callek)]

Comment on attachment 8963979 [details]
Bug 1441353 - part 2: Add beetmover job to publish signed langpacks

Approval Request Comment
[Feature/Bug causing the regression]: Language pack signing
[User impact if declined]: Language packs won't be signed and unable to enable signing required for langpacks
[Is this code covered by automated tests?]: No
[Has the fix been verified in Nightly?]: N/A
[Needs manual test from QE? If yes, steps to reproduce]:  No
[List of other uplifts needed for the feature/fix]: For full support product will need to enable the Langpack Signing pref and get QE testing, but this code enables releng to upload signed langpacks along with our automation.
[Is the change risky?]: Yes
[Why is the change risky/not risky?]: It affects the release process late in the cycle.
[String changes made/needed]: None

Mitigations to release risk:
* If there are failures relating to configuration on specific addons on AMO, and if releng can self-serve fix them, we will do so and rerun affected tasks.
* If there is a minor failure in the addonscript we'll deploy an updated version and rerun the affect task(s)
* If we're unable to accomplish the above within an hour, or if there is in-tree taskgraph bustage from this patch, we back out and instead target a beta next week.

Comment 35

[Ritu Kothari (:ritu) (Inactive, please n-i to RyanVM, jcristau, or pascal)]

I was under the impression that "requiring signed language packs" is a 61 (and ESR60.1) feature. Why does this need to be uplifted to 60?

Comment 36

[Ritu Kothari (:ritu) (Inactive, please n-i to RyanVM, jcristau, or pascal)]

Comment on attachment 8961871 [details]
Bug 1441353 - part 1: Add AMO tasks for signing and uploading

Even though this feature is planned for 61, uplifting it in 60 helps build confidence in our ability to make it in esr60.1. I am hoping the risk mitigations outlined by Callek prevent any negative impact to Beta60.

Comment 37

[Ritu Kothari (:ritu) (Inactive, please n-i to RyanVM, jcristau, or pascal)]

Julien, FYI.

Comment 38

[Mozilla Autoland]

We're sorry, Autoland could not rebase your commits for you automatically. Please manually rebase your commits and try again.

hg error in cmd: hg rebase -s 00803a5eaa095817d70495eb8d3ce871561a110e -d 67c06be1e147: rebasing 457592:00803a5eaa09 "Bug 1441353 - part 1: Add AMO tasks for signing and uploading r=Callek"
merging taskcluster/docs/kinds.rst
warning: conflicts while merging taskcluster/docs/kinds.rst! (edit, then use 'hg resolve --mark')
unresolved conflicts (see hg resolve, then hg rebase --continue)

Comment 39

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/mozilla-inbound/rev/fae9885d7b37
part 1: Add AMO tasks for signing and uploading r=Callek
https://hg.mozilla.org/integration/mozilla-inbound/rev/a7f18f20fab1
part 2: Add beetmover job to publish signed langpacks r=Callek

Comment 40

[Justin Wood (:Callek)]

https://hg.mozilla.org/releases/mozilla-beta/rev/d0b45d0fc8ccdcedaeedb6d1b945bc8da15603e1
https://hg.mozilla.org/releases/mozilla-beta/rev/e34ed4b609e54848bb609d6be64d7300272c97f1

Comment 41

[Justin Wood (:Callek)]

Attachment: Fix beta decision task bustage

Comment 42

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/mozilla-inbound/rev/69d69e8eccd6
Fix beta decision bustage. r=aki a=bustage on a CLOSED TREE

Comment 43

[Justin Wood (:Callek)]

https://hg.mozilla.org/releases/mozilla-beta/rev/5a573ad73b78509ff92daed4d161fb9244fb8dc1

Comment 44

[Raul Gurzau (:RaulG)]

https://hg.mozilla.org/mozilla-central/rev/fae9885d7b37
https://hg.mozilla.org/mozilla-central/rev/a7f18f20fab1
https://hg.mozilla.org/mozilla-central/rev/69d69e8eccd6

Comment 45

[Intermittent Failures Robot]

6 failures in 551 pushes (0.011 failures/push) were associated with this bug in the last 7 days.    

Repository breakdown:
* mozilla-release: 6

Platform breakdown:
* linux64: 5
* osx-cross: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1441353&startday=2018-04-30&endday=2018-05-06&tree=trunk