Create a scriptworker for submission to and fetching from AMO

RESOLVED FIXED

Status

enhancement
RESOLVED FIXED
Last year
Last year

People

(Reporter: Callek, Assigned: Callek)

Tracking

unspecified
Dependency tree / graph

Firefox Tracking Flags

(firefox60 fixed, firefox61 fixed)

Details

Attachments

(5 attachments)

We need to submit addons to AMO for signing and get the signed xpi back for language packs.
Depends on: 1445000
Depends on: 1450104
I added to [1]:
* queue:create-task:low:scriptworker-prov-v1/addon-dev
* project:releng:amo:staging

I also added to [2]:
* queue:create-task:highest:scriptworker-prov-v1/addon-v1
* project:releng:amo:production

[1] https://tools.taskcluster.net/auth/roles/moz-tree%3Alevel%3A1%3Agecko
[2] https://tools.taskcluster.net/auth/roles/moz-tree%3Alevel%3A3%3Agecko
Depends on: 1451315
Comment on attachment 8964918 [details]
Bug 1441353 - Add addon_scriptworker instances

https://reviewboard.mozilla.org/r/233660/#review240270

::: manifests/moco-nodes.pp
(Diff revision 2)
>  # Loaner for dividehex; bug 1445842 and 1447766
>  node 'ds-test1.srv.releng.mdc2.mozilla.com' {
>      $aspects = [ 'low-security' ]
>      include toplevel::server
>  }
> -

EOF change?

::: modules/addon_scriptworker/manifests/init.pp:94
(Diff revision 2)
> +
> +    $config_content = $addon_scriptworker::settings::script_config_content
> +    file {
> +        $addon_scriptworker::settings::script_config:
> +            require => Python35::Virtualenv[$addon_scriptworker::settings::root],
> +            content => inline_template("<%- require 'json' -%><%= JSON.pretty_generate(@config_content) %>");

TIL
Attachment #8964918 - Flags: review?(bugspam.Callek) → review+
Comment on attachment 8964918 [details]
Bug 1441353 - Add addon_scriptworker instances

https://reviewboard.mozilla.org/r/233660/#review240272

I should note, I mostly skimmed the code here.
Comment on attachment 8963979 [details]
Bug 1441353 - part 2: Add beetmover job to publish signed langpacks

https://reviewboard.mozilla.org/r/232798/#review240912


Code analysis found 3 defects in this patch:
 - 3 defects found by mozlint

You can run this analysis locally with:
 - `./mach lint path/to/file` (JS/Python)


If you see a problem in this automated review, please report it here: http://bit.ly/2y9N9Vx


::: taskcluster/taskgraph/transforms/release_beetmover_signed_addons.py:86
(Diff revision 3)
> +        resolve_keyed_by(
> +            job, 'worker-type', item_name=job['label'], project=config.params['project']
> +        )
> +        yield job
> +
> +@transforms.add

Error: Expected 2 blank lines, found 1 [flake8: E302]

::: taskcluster/taskgraph/transforms/release_beetmover_signed_addons.py:110
(Diff revision 3)
> +            locales='/'.join(job['attributes']['chunk_locales']),
> +            platform=job['attributes']['build_platform']
> +        )
> +
> +
> +        job['scopes'] = [

Error: Too many blank lines (2) [flake8: E303]

::: taskcluster/taskgraph/transforms/release_beetmover_signed_addons.py:176
(Diff revision 3)
> +            platform_job = copy.deepcopy(job)
> +
> +            platform_job['attributes']['build_platform'] = platform
> +            platform_job['label'] = job['label'].replace('linux64', platform)
> +            platform_job['description'] = job['description'].replace('linux64', platform)
> +            platform_job['treeherder']['platform'] = platform_job['treeherder']['platform'].replace('linux64', platform)

Error: Line too long (120 > 99 characters) [flake8: E501]
Comment on attachment 8961871 [details]
Bug 1441353 - part 1: Add AMO tasks for signing and uploading

https://reviewboard.mozilla.org/r/230698/#review241012

::: taskcluster/taskgraph/transforms/release_sign_and_push_langpacks.py:5
(Diff revision 7)
> +# This Source Code Form is subject to the terms of the Mozilla Public
> +# License, v. 2.0. If a copy of the MPL was not distributed with this
> +# file, You can obtain one at http://mozilla.org/MPL/2.0/.
> +"""
> +Transform the beetmover task into an actual task description.

nit: docstring message

::: taskcluster/taskgraph/transforms/release_sign_and_push_langpacks.py:49
(Diff revision 7)
> +def set_label(config, jobs):
> +    for job in jobs:
> +        label = 'sign-and-push-langpacks-{}'.format(job['dependent-task'].label)
> +        job['label'] = label
> +
> +        yield job

I wonder if the name_sanity transform could help here (not a blocker)

::: taskcluster/taskgraph/transforms/release_sign_and_push_langpacks.py:73
(Diff revision 7)
> +        if build_platform == 'linux64-nightly':
> +            yield job
> +        elif build_platform == 'macosx64-nightly' and \
> +                'ja-JP-mac' in dep_job_attributes.get('chunk_locales', ['en-US']):
> +            # Other locales of the same job shouldn't be processed
> +            dep_job_attributes['chunk_locales'] = ['ja-JP-mac']

I didn't confirm yet, but I'd want to be sure this doesn't mutate an object on the osx l10n job itself (as in, we don't end up stripping out other locales from signing/beetmover/etc for normal-l10n, possibly ordering-dependent)

::: taskcluster/taskgraph/transforms/release_sign_and_push_langpacks.py:104
(Diff revision 7)
> +        treeherder = job.get('treeherder', {})
> +        treeherder.setdefault('symbol', 'langpack(SnP{})'.format(attributes.get('l10n_chunk', '')))
> +        dep_th_platform = dep_job.task.get('extra', {}).get(
> +            'treeherder', {}).get('machine', {}).get('platform', '')
> +        treeherder.setdefault('platform', '{}/opt'.format(dep_th_platform))
> +        treeherder.setdefault('tier', 2)

Since this is release blocking, tier 1?

::: taskcluster/taskgraph/transforms/release_sign_and_push_langpacks.py:111
(Diff revision 7)
> +
> +        dependent_kind = str(dep_job.kind)
> +        dependencies = {dependent_kind: dep_job.label}
> +
> +        job['attributes'] = copy_attributes_from_dependent_job(dep_job)
> +        job['attributes']['chunk_locales'] = dep_job.attributes.get('chunk_locales', ['en-US'])

This is where we might want to strip OSX locales down to just ja-JP-mac (on the one matching task), to be safe.
Attachment #8961871 - Flags: review?(bugspam.Callek) → review+
Comment on attachment 8961871 [details]
Bug 1441353 - part 1: Add AMO tasks for signing and uploading

https://reviewboard.mozilla.org/r/230698/#review241012

> I wonder if the name_sanity transform could help here (not a blocker)

Could be. I'm doing nasty stuff with label. That would require more clean up. I'd prefer to handle it in a follow up.

> This is where we might want to strip OSX locales down to just ja-JP-mac (on the one matching task), to be safe.

Great call. This was hiding a bug that I didn't catch before. We didn't have ja-JP-mac uploaded. This is now fixed.
Comment on attachment 8963979 [details]
Bug 1441353 - part 2: Add beetmover job to publish signed langpacks

https://reviewboard.mozilla.org/r/232798/#review241018

Please also take care of reviewbot's style concerns (except line length) and we're good.

::: taskcluster/taskgraph/transforms/beetmover_repackage.py
(Diff revision 3)
>      "target.web-platform.tests.tar.gz",
>      "target.xpcshell.tests.zip",
>      "target_info.txt",
>      "target.jsshell.zip",
>      "mozharness.zip",
> -    "target.langpack.xpi",

devedition likely still needs the en-US .xpi uploaded.

::: taskcluster/taskgraph/transforms/beetmover_repackage.py:75
(Diff revision 3)
>      r'^win(32|64)(|-devedition)-nightly$':
>          _DESKTOP_UPSTREAM_ARTIFACTS_UNSIGNED_EN_US + [
>              "host/bin/mar.exe",
>              "host/bin/mbsdiff.exe",
>          ],
> -    r'^(linux(|64)|macosx64|win(32|64))(|-devedition)-nightly-l10n$':
> +    r'^(linux(|64)|macosx64|win(32|64))(|-devedition)-nightly-l10n$': [],

Hrm, this breaks devedition, right? (we still want to publish langpacks for deved, just not yet signed)

::: taskcluster/taskgraph/transforms/release_beetmover_signed_addons.py:185
(Diff revisions 3 - 4)
> +
> +                yield platform_job
> +
>  
> +def _strip_ja_data_from_linux_job(platform_job):
> +    platform_job['description'] = platform_job['description'].replace('ja/', '')

This could use a comment explaining why we do this... (non blocking)

::: taskcluster/taskgraph/transforms/release_sign_and_push_langpacks.py:104
(Diff revision 3)
>          treeherder = job.get('treeherder', {})
>          treeherder.setdefault('symbol', 'langpack(SnP{})'.format(attributes.get('l10n_chunk', '')))
>          dep_th_platform = dep_job.task.get('extra', {}).get(
>              'treeherder', {}).get('machine', {}).get('platform', '')
>          treeherder.setdefault('platform', '{}/opt'.format(dep_th_platform))
> -        treeherder.setdefault('tier', 2)
> +        treeherder.setdefault('tier', 1)

huh how is this switching back to tier 2?

::: taskcluster/taskgraph/transforms/release_sign_and_push_langpacks.py:114
(Diff revision 3)
>  
>          job['description'] = job['description'].format(
>              locales='/'.join(job['attributes']['chunk_locales']),
>          )
>  
> -        job['dependencies'] = dependencies
> +        job['dependencies'] = {

in fact this file looks funny...

::: taskcluster/taskgraph/transforms/task.py:464
(Diff revision 3)
> -        Required('max-run-time'): int,
> +        Required('max-run-time', default=600): int,
>  
>          # locale key, if this is a locale beetmover job
>          Optional('locale'): basestring,
>  
> -        Required('release-properties'): {
> +        Optional('release-properties'): {

This is the one change in this patch I don't claim to know anything about.
Attachment #8963979 - Flags: review?(bugspam.Callek) → review+
Comment on attachment 8963979 [details]
Bug 1441353 - part 2: Add beetmover job to publish signed langpacks

https://reviewboard.mozilla.org/r/232798/#review241048

::: taskcluster/taskgraph/transforms/beetmover_repackage.py:75
(Diff revision 4)
>      r'^win(32|64)(|-devedition)-nightly$':
>          _DESKTOP_UPSTREAM_ARTIFACTS_UNSIGNED_EN_US + [
>              "host/bin/mar.exe",
>              "host/bin/mbsdiff.exe",
>          ],
> -    r'^(linux(|64)|macosx64|win(32|64))(|-devedition)-nightly-l10n$':
> +    r'^(linux(|64)|macosx64|win(32|64))(|-devedition)-nightly-l10n$': [],

still have the devedition concern here...
Comment on attachment 8963979 [details]
Bug 1441353 - part 2: Add beetmover job to publish signed langpacks

https://reviewboard.mozilla.org/r/232798/#review241018

> This is the one change in this patch I don't claim to know anything about.

That was an artifact of an old try of mine. Thank you for calling it out.
Depends on: 1453041
Comment on attachment 8964918 [details]
Bug 1441353 - Add addon_scriptworker instances

https://reviewboard.mozilla.org/r/233660/#review240270

> EOF change?

Ah crap, I landed the patch whi the EOF change. I'll fix it next time.
addonworker-dev-1 and addonworker-1 are now both attached to the production environment.
Depends on: 1453240
Comment on attachment 8961871 [details]
Bug 1441353 - part 1: Add AMO tasks for signing and uploading

Approval Request Comment
[Feature/Bug causing the regression]: Language pack signing
[User impact if declined]: Language packs won't be signed and unable to enable signing required for langpacks
[Is this code covered by automated tests?]: No
[Has the fix been verified in Nightly?]: N/A
[Needs manual test from QE? If yes, steps to reproduce]:  No
[List of other uplifts needed for the feature/fix]: For full support product will need to enable the Langpack Signing pref and get QE testing, but this code enables releng to upload signed langpacks along with our automation.
[Is the change risky?]: Yes
[Why is the change risky/not risky?]: It affects the release process late in the cycle.
[String changes made/needed]: None

Mitigations to release risk:
* If there are failures relating to configuration on specific addons on AMO, and if releng can self-serve fix them, we will do so and rerun affected tasks.
* If there is a minor failure in the addonscript we'll deploy an updated version and rerun the affect task(s)
* If we're unable to accomplish the above within an hour, or if there is in-tree taskgraph bustage from this patch, we back out and instead target a beta next week.
Attachment #8961871 - Flags: approval-mozilla-beta?
Comment on attachment 8963979 [details]
Bug 1441353 - part 2: Add beetmover job to publish signed langpacks

Approval Request Comment
[Feature/Bug causing the regression]: Language pack signing
[User impact if declined]: Language packs won't be signed and unable to enable signing required for langpacks
[Is this code covered by automated tests?]: No
[Has the fix been verified in Nightly?]: N/A
[Needs manual test from QE? If yes, steps to reproduce]:  No
[List of other uplifts needed for the feature/fix]: For full support product will need to enable the Langpack Signing pref and get QE testing, but this code enables releng to upload signed langpacks along with our automation.
[Is the change risky?]: Yes
[Why is the change risky/not risky?]: It affects the release process late in the cycle.
[String changes made/needed]: None

Mitigations to release risk:
* If there are failures relating to configuration on specific addons on AMO, and if releng can self-serve fix them, we will do so and rerun affected tasks.
* If there is a minor failure in the addonscript we'll deploy an updated version and rerun the affect task(s)
* If we're unable to accomplish the above within an hour, or if there is in-tree taskgraph bustage from this patch, we back out and instead target a beta next week.
Attachment #8963979 - Flags: approval-mozilla-beta?
I was under the impression that "requiring signed language packs" is a 61 (and ESR60.1) feature. Why does this need to be uplifted to 60?
Comment on attachment 8961871 [details]
Bug 1441353 - part 1: Add AMO tasks for signing and uploading

Even though this feature is planned for 61, uplifting it in 60 helps build confidence in our ability to make it in esr60.1. I am hoping the risk mitigations outlined by Callek prevent any negative impact to Beta60.
Attachment #8961871 - Flags: approval-mozilla-beta? → approval-mozilla-beta+
Attachment #8963979 - Flags: approval-mozilla-beta? → approval-mozilla-beta+
Julien, FYI.
Flags: needinfo?(jcristau)
We're sorry, Autoland could not rebase your commits for you automatically. Please manually rebase your commits and try again.

hg error in cmd: hg rebase -s 00803a5eaa095817d70495eb8d3ce871561a110e -d 67c06be1e147: rebasing 457592:00803a5eaa09 "Bug 1441353 - part 1: Add AMO tasks for signing and uploading r=Callek"
merging taskcluster/docs/kinds.rst
warning: conflicts while merging taskcluster/docs/kinds.rst! (edit, then use 'hg resolve --mark')
unresolved conflicts (see hg resolve, then hg rebase --continue)
Flags: needinfo?(bugspam.Callek)
Pushed by Callek@gmail.com:
https://hg.mozilla.org/integration/mozilla-inbound/rev/fae9885d7b37
part 1: Add AMO tasks for signing and uploading r=Callek
https://hg.mozilla.org/integration/mozilla-inbound/rev/a7f18f20fab1
part 2: Add beetmover job to publish signed langpacks r=Callek
Attachment #8967439 - Flags: review?(aki)
Attachment #8967439 - Flags: review?(aki) → review+
Pushed by Callek@gmail.com:
https://hg.mozilla.org/integration/mozilla-inbound/rev/69d69e8eccd6
Fix beta decision bustage. r=aki a=bustage on a CLOSED TREE
Flags: needinfo?(jcristau)
Component: General Automation → General
You need to log in before you can comment on or make changes to this bug.