1415981 - migrate final verify in tree

Status: RESOLVED FIXED

(Release Engineering :: Release Automation: Other, enhancement)

Comment 1

[[e-mail] (:bhearsum)]

Attachment: in tree final verify

Relatively straightforward - mostly just migrates the releasetasks task definition into the tree. The latest run of this is https://tools.taskcluster.net/groups/QhNYiURETkqk6kEanP5K8A/tasks/OQGer5M0Qv2ItN2A0lLmyg/details, which invokes exactly the same command as a recent beta (https://tools.taskcluster.net/groups/SO7pdJTfTQ2sDyJdm91AVg/tasks/Ft9bHzHgTPSauxRSAs_l0A/details). This will continue to fail until required upstreams are ready and defined.

Comment 2

[[e-mail] (:bhearsum)]

If anyone is picking this up while I'm away, the easiest way to find the isolated code for this in my git branch: https://github.com/mozbhearsum/gecko-dev/compare/final-verify-in-tree?expand=1 -- it's up-to-date (as of now) against maple, and contains only the final verify work.

Comment 3

[Aki Sasaki (not active)]

Comment on attachment 8927465 [details] [diff] [review]
in tree final verify

Thanks! r+ if we specify the default env var below.

> diff --git a/taskcluster/ci/release-final-verify/kind.yml b/taskcluster/ci/release-final-verify/kind.yml
> new file mode 100644
> index 000000000000..e66ba287f3ef
> --- /dev/null
> +++ b/taskcluster/ci/release-final-verify/kind.yml
> @@ -0,0 +1,51 @@
> +# This Source Code Form is subject to the terms of the Mozilla Public
> +# License, v. 2.0. If a copy of the MPL was not distributed with this
> +# file, You can obtain one at http://mozilla.org/MPL/2.0/.
> +
> +loader: taskgraph.loader.transform:loader
> +
> +# TODO: we probably hove some of these?
> +#kind-dependencies:
>
> +transforms:
> +    - taskgraph.transforms.final_verify:transforms
> +    - taskgraph.transforms.task:transforms

It looks like we should depend on release-uptake-monitoring, judging by Fx57.0b9?
https://tools.taskcluster.net/groups/SO7pdJTfTQ2sDyJdm91AVg/tasks/Ft9bHzHgTPSauxRSAs_l0A/details

Once we do that, we should add `release_deps` as a transform.

> +job-defaults:
> +    label: release-final-verify
> +    name: final-verify
> +    description: final verify
> +    run-on-projects: [] # to make sure this never runs as part of CI
> +    worker-type: aws-provisioner-v1/gecko-{level}-b-linux
> +    worker:
> +        implementation: docker-worker
> +        os: linux
> +        docker-image: mozillareleases/python-test-runner@sha256:0729c2e6e7bc0d6a4cbccb2e66a78e1d8e8cbb5e44105d56e3c9c610230ebd69
> +        max-run-time: 7200
> +        retry-exit-status: 1
> +        env:
> +            BUILD_TOOLS_REPO:
> +                by-project:
> +                    jamun: https://hg.mozilla.org/users/stage-ffxbld/tools
> +                    maple: https://hg.mozilla.org/users/asasaki_mozilla.com/tools
> +                    default: https://hg.mozilla.org/build/tools
> +    treeherder:
> +        platform: linux64/opt
> +        kind: test
> +        symbol: Rel(FV)
> +        tier: 1
> +    extra:

I think we can skip `extra:` if there are no contents.

> +jobs:
> +    firefox:
> +        worker:
> +            env:
> +                FINAL_VERIFY_CONFIGS:
> +                    by-project:
> +                        jamun: "beta-firefox-linux.cfg beta-firefox-linux64.cfg beta-firefox-macosx64.cfg beta-firefox-win32.cfg beta-firefox-win64.cfg"
> +                        maple: "beta-firefox-linux.cfg beta-firefox-linux64.cfg beta-firefox-macosx64.cfg beta-firefox-win32.cfg beta-firefox-win64.cfg"
> +                        mozilla-beta: "beta-firefox-linux.cfg beta-firefox-linux64.cfg beta-firefox-macosx64.cfg beta-firefox-win32.cfg beta-firefox-win64.cfg"
> +                        mozilla-release: "release-firefox-linux.cfg release-firefox-linux64.cfg release-firefox-macosx64.cfg release-firefox-win32.cfg release-firefox-win64.cfg"
> +                        mozilla-esr52: "esr-firefox-linux.cfg esr-firefox-linux64.cfg esr-firefox-macosx64.cfg esr-firefox-win32.cfg esr-firefox-win64.cfg"

This breaks m-c and try graph generation because there's no default. I fixed it on maple with a `default: ""`

>      def filter(task):
> -        if task.kind in ('release-update-verify', 'release-buildbot-update-verify'):
> +        if task.kind in ('release-final-verify', 'release-update-verify',
> +                         'release-buildbot-update-verify'):

Hoping the new shipping_phase attribute helps here.

Comment 4

[[e-mail] (:bhearsum)]

I've made some updates to this work, but for the life of my I can't figure out how to get a useful diff out of maple now. The latest changes are:
https://hg.mozilla.org/projects/maple/rev/04ae2470c78eedf0e42052c7956ce7877b4134c3 - which is the nit fix you requested
https://hg.mozilla.org/projects/maple/rev/95b95ae9bca6dda1e117e2e853e86e487cabd088 - which does a few things:
- sets the dependency on release-uptake-monitoring
- moves the shipping phase to publish (i'm not 100% sure if this is correct, but at the very least it should be in the same phase as uptake-monitoring)
- use the correct tag in the tools repo
- the fix for https://bugzilla.mozilla.org/show_bug.cgi?id=1419778, to make it possible to have dependencies between buildbot-bridge and non buildbot-bridge tasks

Aki, how does this look now?

Comment 5

[[e-mail] (:bhearsum)]

Cancelling needinfo for now because I realized that I still need to account for all the differences between beta vs rc vs esr.

Comment 6

[[e-mail] (:bhearsum)]

Attachment: updated in tree final verify

New in this version:
* Set upstreams correctly (release-uptake-monitoring)
* Move to "publish" phase

This won't work for Devedition yet, but should be easy to add. It should work for the "release" channel on RCs, but won't work for the "beta" channel on RCs. That part is going to be complicated, because it doesn't depend on uptake monitoring, and could potentially be a different kind altogether.

Comment 7

[Aki Sasaki (not active)]

Comment on attachment 8932228 [details] [diff] [review]
updated in tree final verify

>diff --git a/taskcluster/ci/release-final-verify/kind.yml b/taskcluster/ci/release-final-verify/kind.yml
>new file mode 100644
>--- /dev/null
>+++ b/taskcluster/ci/release-final-verify/kind.yml
>@@ -0,0 +1,63 @@
>+# This Source Code Form is subject to the terms of the Mozilla Public
>+# License, v. 2.0. If a copy of the MPL was not distributed with this
>+# file, You can obtain one at http://mozilla.org/MPL/2.0/.
>+
>+loader: taskgraph.loader.transform:loader
>+
>+kind-dependencies:
>+    - release-uptake-monitoring
>+
>+transforms:
>+    - taskgraph.transforms.release_deps:transforms
>+    - taskgraph.transforms.final_verify:transforms
>+    - taskgraph.transforms.task:transforms
>+
>+job-defaults:
>+    name: final-verify
>+    run-on-projects: [] # to make sure this never runs as part of CI
>+    worker-type: aws-provisioner-v1/gecko-{level}-b-linux
>+    worker:
>+        implementation: docker-worker
>+        os: linux
>+        docker-image: mozillareleases/python-test-runner@sha256:0729c2e6e7bc0d6a4cbccb2e66a78e1d8e8cbb5e44105d56e3c9c610230ebd69
>+        max-run-time: 7200
>+        retry-exit-status: 1
>+        env:
>+            BUILD_TOOLS_REPO:
>+                by-project:
>+                    jamun: https://hg.mozilla.org/users/stage-ffxbld/tools
>+                    maple: https://hg.mozilla.org/users/asasaki_mozilla.com/tools
>+                    default: https://hg.mozilla.org/build/tools
>+
>+jobs:
>+    firefox:
>+        label: release-final-verify
>+        description: final verify
>+        shipping-phase: publish
>+        shipping-product: firefox
>+        worker:
>+            env:
>+                FINAL_VERIFY_CONFIGS:
>+                    by-project:
>+                        jamun: "beta-firefox-linux.cfg beta-firefox-linux64.cfg beta-firefox-macosx64.cfg beta-firefox-win32.cfg beta-firefox-win64.cfg"
>+                        maple: "beta-firefox-linux.cfg beta-firefox-linux64.cfg beta-firefox-macosx64.cfg beta-firefox-win32.cfg beta-firefox-win64.cfg"
>+                        mozilla-beta: "beta-firefox-linux.cfg beta-firefox-linux64.cfg beta-firefox-macosx64.cfg beta-firefox-win32.cfg beta-firefox-win64.cfg"
>+                        mozilla-release: "release-firefox-linux.cfg release-firefox-linux64.cfg release-firefox-macosx64.cfg release-firefox-win32.cfg release-firefox-win64.cfg"
>+                        mozilla-esr52: "esr-firefox-linux.cfg esr-firefox-linux64.cfg esr-firefox-macosx64.cfg esr-firefox-win32.cfg esr-firefox-win64.cfg"
>+                        default: ""
>+        extra:
>+            product: firefox
>+        treeherder:
>+            platform: linux64/opt
>+            kind: test
>+            symbol: Rel(FV)
>+            tier: 1
>diff --git a/taskcluster/docs/kinds.rst b/taskcluster/docs/kinds.rst
>--- a/taskcluster/docs/kinds.rst
>+++ b/taskcluster/docs/kinds.rst
>@@ -239,6 +243,18 @@ release-bouncer-aliases
> ------------------------------
> Update Bouncers (download.mozilla.org) "latest" aliases.
> 
>+release-final-verify
>+---------------------
>+Verifies the contents and package of release update MARs.
>+
> release-uptake-monitoring
> -------------------------
> Run uptake monitoring for releases.
>diff --git a/taskcluster/taskgraph/target_tasks.py b/taskcluster/taskgraph/target_tasks.py
>--- a/taskcluster/taskgraph/target_tasks.py
>+++ b/taskcluster/taskgraph/target_tasks.py
>diff --git a/taskcluster/taskgraph/transforms/final_verify.py b/taskcluster/taskgraph/transforms/final_verify.py
>new file mode 100644
>--- /dev/null
>+++ b/taskcluster/taskgraph/transforms/final_verify.py
>@@ -0,0 +1,40 @@
>+# This Source Code Form is subject to the terms of the Mozilla Public
>+# License, v. 2.0. If a copy of the MPL was not distributed with this
>+# file, You can obtain one at http://mozilla.org/MPL/2.0/.
>+"""
>+Transform the beetmover task into an actual task description.
>+"""
>+
>+from __future__ import absolute_import, print_function, unicode_literals
>+
>+from taskgraph.transforms.base import TransformSequence
>+from taskgraph.util.schema import resolve_keyed_by
>+from taskgraph.util.scriptworker import get_release_config
>+
>+transforms = TransformSequence()
>+
>+
>[e-mail]
>+def add_command(config, tasks):
>+    for task in tasks:
>+        release_config = get_release_config(config)
>+        release_tag = "{}_{}_RELEASE_RUNTIME".format(
>+            task["extra"]["product"].upper(),
>+            release_config["version"].replace(".", "_")
>+        )
>+
>+        if not task["worker"].get("env"):
>+            task["worker"]["env"] = {}
>+        task["worker"]["command"] = [
>+            "/bin/bash",
>+            "-c",
>+            "hg clone $BUILD_TOOLS_REPO tools && cd tools &&" +
>+            "hg up -r {} && cd release && ".format(
>+                release_tag,
>+            ) +
>+            "./final-verification.sh $FINAL_VERIFY_CONFIGS"
>+        ]
>+        for thing in ("FINAL_VERIFY_CONFIGS", "BUILD_TOOLS_REPO"):
>+            thing = "worker.env.{}".format(thing)
>+            resolve_keyed_by(task, thing, thing, **config.params)
>+        yield task
>diff --git a/taskcluster/taskgraph/transforms/release_deps.py b/taskcluster/taskgraph/transforms/release_deps.py
>--- a/taskcluster/taskgraph/transforms/release_deps.py
>+++ b/taskcluster/taskgraph/transforms/release_deps.py
>@@ -36,6 +36,7 @@ def add_dependencies(config, jobs):
>         dependencies = {}
>         # Add any kind_dependencies_tasks with matching product as dependencies
>         product = _get_product(job)
>+        phase = job.get('shipping-phase')
>         if product is None:
>             continue
> 
>@@ -52,9 +53,16 @@ def add_dependencies(config, jobs):
>                 # Skip old-id
>                 if 'old-id' in dep_task.label:
>                     continue
>+            # We can only depend on tasks within the same phase
>+            if dep_task.attributes.get('shipping_phase') != phase:
>+                continue
>             # Add matching product tasks to deps
>             if _get_product(dep_task.task) == product:
>                 dependencies[dep_task.label] = dep_task.label
>+            # this is dumb, but i don't know how to get beetmover to set
>+            # shipping-product in a way that _get_product recognizes
>+            if dep_task.attributes.get('shipping_product') == product:
>+                dependencies[dep_task.label] = dep_task.label
> 
>         job.setdefault('dependencies', {}).update(dependencies)
> 
>diff --git a/taskcluster/taskgraph/transforms/task.py b/taskcluster/taskgraph/transforms/task.py
>--- a/taskcluster/taskgraph/transforms/task.py
>+++ b/taskcluster/taskgraph/transforms/task.py
>@@ -164,6 +164,8 @@ task_description_schema = Schema({
>             # for non-tier-1 tasks.
>             'build_date',
>         ),
>+
>+        'channel': optionally_keyed_by('project', basestring),
>     },
> 
>     # The `run_on_projects` attribute, defaulting to "all".  This dictates the
>@@ -333,7 +335,10 @@ task_description_schema = Schema({
>         Required('max-run-time'): int,
> 
>         # the exit status code that indicates the task should be retried
>-        Optional('retry-exit-status'): int,
>+        Optional('retry-exit-status'): Any(
>+            int,
>+            [int],
>+        ),
>     }, {
>         Required('implementation'): 'generic-worker',
>         Required('os'): Any('windows', 'macosx'),
>@@ -774,7 +789,10 @@ def build_docker_worker_payload(config, 
>         payload['maxRunTime'] = worker['max-run-time']
> 
>     if 'retry-exit-status' in worker:
>-        payload['onExitStatus'] = {'retry': [worker['retry-exit-status']]}
>+        if isinstance(worker['retry-exit-status'], int):
>+            payload['onExitStatus'] = {'retry': [worker['retry-exit-status']]}
>+        elif isinstance(worker['retry-exit-status'], list):
>+            payload['onExitStatus'] = {'retry': worker['retry-exit-status']}
> 
>     if 'artifacts' in worker:
>         artifacts = {}

Comment 8

[[e-mail] (:bhearsum)]

Attachment: RC support

I built this on top of the RC support for final verify in bug 1398793. As with that patch, this only runs for desktop_release_type == rc. I ended up using a different kind because the dependencies and phase are different than the other final-verify kind (this one runs in "promote", and depends on beetmover/balrog instead of running in "push" and depending on uptake monitoring).

Comment 9

[Aki Sasaki (not active)]

Comment on attachment 8933691 [details] [diff] [review]
RC support

Thanks!
Same here, let's remove the extra: product: lines.

Comment 10

[[e-mail] (:bhearsum)]

The only thing left to do here is verify that RC support works.

Comment 11

[[e-mail] (:bhearsum)]

Attachment: use correct secondary final verify configs

Comment 12

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/mozilla-inbound/rev/0cdbe30a073a
use correct configs in secondary final verify. r=aki

Comment 13

[Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)]

https://hg.mozilla.org/mozilla-central/rev/0cdbe30a073a