Status

enhancement
RESOLVED FIXED
2 years ago
a year ago

People

(Reporter: bhearsum, Assigned: bhearsum)

Tracking

unspecified
Dependency tree / graph

Firefox Tracking Flags

(firefox59 fixed)

Details

Attachments

(3 attachments, 1 obsolete attachment)

No description provided.
Assignee

Updated

2 years ago
Assignee: nobody → bhearsum
Assignee

Updated

2 years ago
Blocks: 1397773
No longer blocks: 1397762
Posted patch in tree final verify (obsolete) — Splinter Review
Relatively straightforward - mostly just migrates the releasetasks task definition into the tree. The latest run of this is https://tools.taskcluster.net/groups/QhNYiURETkqk6kEanP5K8A/tasks/OQGer5M0Qv2ItN2A0lLmyg/details, which invokes exactly the same command as a recent beta (https://tools.taskcluster.net/groups/SO7pdJTfTQ2sDyJdm91AVg/tasks/Ft9bHzHgTPSauxRSAs_l0A/details). This will continue to fail until required upstreams are ready and defined.
Attachment #8927465 - Flags: review?(aki)
If anyone is picking this up while I'm away, the easiest way to find the isolated code for this in my git branch: https://github.com/mozbhearsum/gecko-dev/compare/final-verify-in-tree?expand=1 -- it's up-to-date (as of now) against maple, and contains only the final verify work.
Comment on attachment 8927465 [details] [diff] [review]
in tree final verify

Thanks! r+ if we specify the default env var below.

> diff --git a/taskcluster/ci/release-final-verify/kind.yml b/taskcluster/ci/release-final-verify/kind.yml
> new file mode 100644
> index 000000000000..e66ba287f3ef
> --- /dev/null
> +++ b/taskcluster/ci/release-final-verify/kind.yml
> @@ -0,0 +1,51 @@
> +# This Source Code Form is subject to the terms of the Mozilla Public
> +# License, v. 2.0. If a copy of the MPL was not distributed with this
> +# file, You can obtain one at http://mozilla.org/MPL/2.0/.
> +
> +loader: taskgraph.loader.transform:loader
> +
> +# TODO: we probably hove some of these?
> +#kind-dependencies:
>
> +transforms:
> +    - taskgraph.transforms.final_verify:transforms
> +    - taskgraph.transforms.task:transforms

It looks like we should depend on release-uptake-monitoring, judging by Fx57.0b9?
https://tools.taskcluster.net/groups/SO7pdJTfTQ2sDyJdm91AVg/tasks/Ft9bHzHgTPSauxRSAs_l0A/details

Once we do that, we should add `release_deps` as a transform.

> +job-defaults:
> +    label: release-final-verify
> +    name: final-verify
> +    description: final verify
> +    run-on-projects: [] # to make sure this never runs as part of CI
> +    worker-type: aws-provisioner-v1/gecko-{level}-b-linux
> +    worker:
> +        implementation: docker-worker
> +        os: linux
> +        docker-image: mozillareleases/python-test-runner@sha256:0729c2e6e7bc0d6a4cbccb2e66a78e1d8e8cbb5e44105d56e3c9c610230ebd69
> +        max-run-time: 7200
> +        retry-exit-status: 1
> +        env:
> +            BUILD_TOOLS_REPO:
> +                by-project:
> +                    jamun: https://hg.mozilla.org/users/stage-ffxbld/tools
> +                    maple: https://hg.mozilla.org/users/asasaki_mozilla.com/tools
> +                    default: https://hg.mozilla.org/build/tools
> +    treeherder:
> +        platform: linux64/opt
> +        kind: test
> +        symbol: Rel(FV)
> +        tier: 1
> +    extra:

I think we can skip `extra:` if there are no contents.

> +jobs:
> +    firefox:
> +        worker:
> +            env:
> +                FINAL_VERIFY_CONFIGS:
> +                    by-project:
> +                        jamun: "beta-firefox-linux.cfg beta-firefox-linux64.cfg beta-firefox-macosx64.cfg beta-firefox-win32.cfg beta-firefox-win64.cfg"
> +                        maple: "beta-firefox-linux.cfg beta-firefox-linux64.cfg beta-firefox-macosx64.cfg beta-firefox-win32.cfg beta-firefox-win64.cfg"
> +                        mozilla-beta: "beta-firefox-linux.cfg beta-firefox-linux64.cfg beta-firefox-macosx64.cfg beta-firefox-win32.cfg beta-firefox-win64.cfg"
> +                        mozilla-release: "release-firefox-linux.cfg release-firefox-linux64.cfg release-firefox-macosx64.cfg release-firefox-win32.cfg release-firefox-win64.cfg"
> +                        mozilla-esr52: "esr-firefox-linux.cfg esr-firefox-linux64.cfg esr-firefox-macosx64.cfg esr-firefox-win32.cfg esr-firefox-win64.cfg"

This breaks m-c and try graph generation because there's no default. I fixed it on maple with a `default: ""`

>      def filter(task):
> -        if task.kind in ('release-update-verify', 'release-buildbot-update-verify'):
> +        if task.kind in ('release-final-verify', 'release-update-verify',
> +                         'release-buildbot-update-verify'):

Hoping the new shipping_phase attribute helps here.
Attachment #8927465 - Flags: review?(aki) → review+
Assignee

Updated

2 years ago
Depends on: 1419778
I've made some updates to this work, but for the life of my I can't figure out how to get a useful diff out of maple now. The latest changes are:
https://hg.mozilla.org/projects/maple/rev/04ae2470c78eedf0e42052c7956ce7877b4134c3 - which is the nit fix you requested
https://hg.mozilla.org/projects/maple/rev/95b95ae9bca6dda1e117e2e853e86e487cabd088 - which does a few things:
- sets the dependency on release-uptake-monitoring
- moves the shipping phase to publish (i'm not 100% sure if this is correct, but at the very least it should be in the same phase as uptake-monitoring)
- use the correct tag in the tools repo
- the fix for https://bugzilla.mozilla.org/show_bug.cgi?id=1419778, to make it possible to have dependencies between buildbot-bridge and non buildbot-bridge tasks

Aki, how does this look now?
Flags: needinfo?(aki)
Cancelling needinfo for now because I realized that I still need to account for all the differences between beta vs rc vs esr.
Flags: needinfo?(aki)
New in this version:
* Set upstreams correctly (release-uptake-monitoring)
* Move to "publish" phase

This won't work for Devedition yet, but should be easy to add. It should work for the "release" channel on RCs, but won't work for the "beta" channel on RCs. That part is going to be complicated, because it doesn't depend on uptake monitoring, and could potentially be a different kind altogether.
Attachment #8927465 - Attachment is obsolete: true
Attachment #8932228 - Flags: feedback?(aki)
Comment on attachment 8932228 [details] [diff] [review]
updated in tree final verify

>diff --git a/taskcluster/ci/release-final-verify/kind.yml b/taskcluster/ci/release-final-verify/kind.yml
>new file mode 100644
>--- /dev/null
>+++ b/taskcluster/ci/release-final-verify/kind.yml
>@@ -0,0 +1,63 @@
>+# This Source Code Form is subject to the terms of the Mozilla Public
>+# License, v. 2.0. If a copy of the MPL was not distributed with this
>+# file, You can obtain one at http://mozilla.org/MPL/2.0/.
>+
>+loader: taskgraph.loader.transform:loader
>+
>+kind-dependencies:
>+    - release-uptake-monitoring
>+
>+transforms:
>+    - taskgraph.transforms.release_deps:transforms
>+    - taskgraph.transforms.final_verify:transforms
>+    - taskgraph.transforms.task:transforms
>+
>+job-defaults:
>+    name: final-verify
>+    run-on-projects: [] # to make sure this never runs as part of CI
>+    worker-type: aws-provisioner-v1/gecko-{level}-b-linux
>+    worker:
>+        implementation: docker-worker
>+        os: linux
>+        docker-image: mozillareleases/python-test-runner@sha256:0729c2e6e7bc0d6a4cbccb2e66a78e1d8e8cbb5e44105d56e3c9c610230ebd69
>+        max-run-time: 7200
>+        retry-exit-status: 1
>+        env:
>+            BUILD_TOOLS_REPO:
>+                by-project:
>+                    jamun: https://hg.mozilla.org/users/stage-ffxbld/tools
>+                    maple: https://hg.mozilla.org/users/asasaki_mozilla.com/tools
>+                    default: https://hg.mozilla.org/build/tools
>+
>+jobs:
>+    firefox:
>+        label: release-final-verify
>+        description: final verify
>+        shipping-phase: publish
>+        shipping-product: firefox
>+        worker:
>+            env:
>+                FINAL_VERIFY_CONFIGS:
>+                    by-project:
>+                        jamun: "beta-firefox-linux.cfg beta-firefox-linux64.cfg beta-firefox-macosx64.cfg beta-firefox-win32.cfg beta-firefox-win64.cfg"
>+                        maple: "beta-firefox-linux.cfg beta-firefox-linux64.cfg beta-firefox-macosx64.cfg beta-firefox-win32.cfg beta-firefox-win64.cfg"
>+                        mozilla-beta: "beta-firefox-linux.cfg beta-firefox-linux64.cfg beta-firefox-macosx64.cfg beta-firefox-win32.cfg beta-firefox-win64.cfg"
>+                        mozilla-release: "release-firefox-linux.cfg release-firefox-linux64.cfg release-firefox-macosx64.cfg release-firefox-win32.cfg release-firefox-win64.cfg"
>+                        mozilla-esr52: "esr-firefox-linux.cfg esr-firefox-linux64.cfg esr-firefox-macosx64.cfg esr-firefox-win32.cfg esr-firefox-win64.cfg"
>+                        default: ""
>+        extra:
>+            product: firefox
>+        treeherder:
>+            platform: linux64/opt
>+            kind: test
>+            symbol: Rel(FV)
>+            tier: 1
>diff --git a/taskcluster/docs/kinds.rst b/taskcluster/docs/kinds.rst
>--- a/taskcluster/docs/kinds.rst
>+++ b/taskcluster/docs/kinds.rst
>@@ -239,6 +243,18 @@ release-bouncer-aliases
> ------------------------------
> Update Bouncers (download.mozilla.org) "latest" aliases.
> 
>+release-final-verify
>+---------------------
>+Verifies the contents and package of release update MARs.
>+
> release-uptake-monitoring
> -------------------------
> Run uptake monitoring for releases.
>diff --git a/taskcluster/taskgraph/target_tasks.py b/taskcluster/taskgraph/target_tasks.py
>--- a/taskcluster/taskgraph/target_tasks.py
>+++ b/taskcluster/taskgraph/target_tasks.py
>diff --git a/taskcluster/taskgraph/transforms/final_verify.py b/taskcluster/taskgraph/transforms/final_verify.py
>new file mode 100644
>--- /dev/null
>+++ b/taskcluster/taskgraph/transforms/final_verify.py
>@@ -0,0 +1,40 @@
>+# This Source Code Form is subject to the terms of the Mozilla Public
>+# License, v. 2.0. If a copy of the MPL was not distributed with this
>+# file, You can obtain one at http://mozilla.org/MPL/2.0/.
>+"""
>+Transform the beetmover task into an actual task description.
>+"""
>+
>+from __future__ import absolute_import, print_function, unicode_literals
>+
>+from taskgraph.transforms.base import TransformSequence
>+from taskgraph.util.schema import resolve_keyed_by
>+from taskgraph.util.scriptworker import get_release_config
>+
>+transforms = TransformSequence()
>+
>+
>+@transforms.add
>+def add_command(config, tasks):
>+    for task in tasks:
>+        release_config = get_release_config(config)
>+        release_tag = "{}_{}_RELEASE_RUNTIME".format(
>+            task["extra"]["product"].upper(),
>+            release_config["version"].replace(".", "_")
>+        )
>+
>+        if not task["worker"].get("env"):
>+            task["worker"]["env"] = {}
>+        task["worker"]["command"] = [
>+            "/bin/bash",
>+            "-c",
>+            "hg clone $BUILD_TOOLS_REPO tools && cd tools &&" +
>+            "hg up -r {} && cd release && ".format(
>+                release_tag,
>+            ) +
>+            "./final-verification.sh $FINAL_VERIFY_CONFIGS"
>+        ]
>+        for thing in ("FINAL_VERIFY_CONFIGS", "BUILD_TOOLS_REPO"):
>+            thing = "worker.env.{}".format(thing)
>+            resolve_keyed_by(task, thing, thing, **config.params)
>+        yield task
>diff --git a/taskcluster/taskgraph/transforms/release_deps.py b/taskcluster/taskgraph/transforms/release_deps.py
>--- a/taskcluster/taskgraph/transforms/release_deps.py
>+++ b/taskcluster/taskgraph/transforms/release_deps.py
>@@ -36,6 +36,7 @@ def add_dependencies(config, jobs):
>         dependencies = {}
>         # Add any kind_dependencies_tasks with matching product as dependencies
>         product = _get_product(job)
>+        phase = job.get('shipping-phase')
>         if product is None:
>             continue
> 
>@@ -52,9 +53,16 @@ def add_dependencies(config, jobs):
>                 # Skip old-id
>                 if 'old-id' in dep_task.label:
>                     continue
>+            # We can only depend on tasks within the same phase
>+            if dep_task.attributes.get('shipping_phase') != phase:
>+                continue
>             # Add matching product tasks to deps
>             if _get_product(dep_task.task) == product:
>                 dependencies[dep_task.label] = dep_task.label
>+            # this is dumb, but i don't know how to get beetmover to set
>+            # shipping-product in a way that _get_product recognizes
>+            if dep_task.attributes.get('shipping_product') == product:
>+                dependencies[dep_task.label] = dep_task.label
> 
>         job.setdefault('dependencies', {}).update(dependencies)
> 
>diff --git a/taskcluster/taskgraph/transforms/task.py b/taskcluster/taskgraph/transforms/task.py
>--- a/taskcluster/taskgraph/transforms/task.py
>+++ b/taskcluster/taskgraph/transforms/task.py
>@@ -164,6 +164,8 @@ task_description_schema = Schema({
>             # for non-tier-1 tasks.
>             'build_date',
>         ),
>+
>+        'channel': optionally_keyed_by('project', basestring),
>     },
> 
>     # The `run_on_projects` attribute, defaulting to "all".  This dictates the
>@@ -333,7 +335,10 @@ task_description_schema = Schema({
>         Required('max-run-time'): int,
> 
>         # the exit status code that indicates the task should be retried
>-        Optional('retry-exit-status'): int,
>+        Optional('retry-exit-status'): Any(
>+            int,
>+            [int],
>+        ),
>     }, {
>         Required('implementation'): 'generic-worker',
>         Required('os'): Any('windows', 'macosx'),
>@@ -774,7 +789,10 @@ def build_docker_worker_payload(config, 
>         payload['maxRunTime'] = worker['max-run-time']
> 
>     if 'retry-exit-status' in worker:
>-        payload['onExitStatus'] = {'retry': [worker['retry-exit-status']]}
>+        if isinstance(worker['retry-exit-status'], int):
>+            payload['onExitStatus'] = {'retry': [worker['retry-exit-status']]}
>+        elif isinstance(worker['retry-exit-status'], list):
>+            payload['onExitStatus'] = {'retry': worker['retry-exit-status']}
> 
>     if 'artifacts' in worker:
>         artifacts = {}
Attachment #8932228 - Flags: feedback?(aki) → feedback+
Posted patch RC supportSplinter Review
I built this on top of the RC support for final verify in bug 1398793. As with that patch, this only runs for desktop_release_type == rc. I ended up using a different kind because the dependencies and phase are different than the other final-verify kind (this one runs in "promote", and depends on beetmover/balrog instead of running in "push" and depending on uptake monitoring).
Attachment #8933691 - Flags: feedback?(aki)
Comment on attachment 8933691 [details] [diff] [review]
RC support

Thanks!
Same here, let's remove the extra: product: lines.
Attachment #8933691 - Flags: feedback?(aki) → feedback+
The only thing left to do here is verify that RC support works.
Attachment #8940838 - Flags: review?(aki) → review+
Attachment #8940838 - Flags: checked-in+

Comment 12

a year ago
Pushed by bhearsum@mozilla.com:
https://hg.mozilla.org/integration/mozilla-inbound/rev/0cdbe30a073a
use correct configs in secondary final verify. r=aki
https://hg.mozilla.org/mozilla-central/rev/0cdbe30a073a
Status: NEW → RESOLVED
Last Resolved: a year ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.