1333155 - convert buildbot addon devel Linux64 builds to taskcluster

Status: RESOLVED FIXED

(Firefox Build System :: Task Configuration, task)

Description

[Kim Moir [:kmoir] ET]

I think they just run on 
'mozilla-beta', 'mozilla-release', 'mozilla-esr45'

Comment 1

[Kim Moir [:kmoir] ET]

Attachment: bug1333155.patch

https://treeherder.mozilla.org/#/jobs?repo=try&revision=fbda5cfbdeb38d22c5967cad5f7dc156a6531f6f

https://public-artifacts.taskcluster.net/YRC_qeqcS_ytPJBIiyW81g/0/public/full-task-graph.json

Comment 2

[Mihai Tabara [:mtabara]⌚️GMT]

Comment on attachment 8830912 [details] [diff] [review]
bug1333155.patch

># HG changeset patch
># User Kim Moir <[e-mail]>
># Parent  84d3a0578ffe449adcb7847e2d34312efbe4e5cf
>try: -b o -p linux64 -u none -t none
>
>diff --git a/taskcluster/ci/build/linux.yml b/taskcluster/ci/build/linux.yml
>--- a/taskcluster/ci/build/linux.yml
>+++ b/taskcluster/ci/build/linux.yml
>+        custom-build-variant-cfg: add-on-devel
>+        tooltool-downloads: public
>+        need-xvfb: true
>+    run-on-projects: [ 'mozilla-beta', 'mozilla-release' ]

I may have gotten this wrong but shouldn't we add `mozilla-esr45` as well as per [1]?
[1]: https://bugzilla.mozilla.org/show_bug.cgi?id=1333155#c0

Comment 3

[Kim Moir [:kmoir] ET]

Attachment: addon-devel-2.patch

good catch Mihai, I have updated the patch

Comment 4

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/mozilla-inbound/rev/32cba5ea7f57
convert buildbot addon devel Linux64 builds to taskcluster r=mtabara

Comment 5

[Wes Kocher (:KWierso) (Not reading bugmail; email directly if needed)]

https://hg.mozilla.org/mozilla-central/rev/32cba5ea7f57

Comment 6

[Kim Moir [:kmoir] ET]

I still have to uplift this to m-b and m-r so I can disable the buildbot builds.

Comment 7

[Kim Moir [:kmoir] ET]

Attachment: wip patches for windows addon

Comment 8

[Kim Moir [:kmoir] ET]

Attachment: wip patches for windows/macosx addon

Comment 9

[Kim Moir [:kmoir] ET]

Attachment: patches for macosx/windows addon builds on tc

https://treeherder.mozilla.org/#/jobs?repo=try&revision=636d565d44161f85c4a567a0b2cfdc5d7156084c&selectedJob=74336265

https://public-artifacts.taskcluster.net/QSHjBxo8SxGKvvGlpsjOcA/0/public/full-task-graph.json

Comment 10

[Mihai Tabara [:mtabara]⌚️GMT]

Comment on attachment 8833337 [details] [diff] [review]
patches for macosx/windows addon builds on tc

># HG changeset patch
># User Kim Moir <[e-mail]>
># Parent  dcfaa9b629bb565db62fec9cc4233c778f6c7e88
>try: -b o -p win64 -u none -t none
>
>diff --git a/taskcluster/ci/build/macosx.yml b/taskcluster/ci/build/macosx.yml
>--- a/taskcluster/ci/build/macosx.yml
>+++ b/taskcluster/ci/build/macosx.yml
>@@ -39,8 +39,33 @@ macosx64/opt:
>+       actions: [get-secrets build generate-build-stats update]
>+       config:
>+            - builds/releng_base_mac_64_cross_builds.py
>+            - balrog/production.py

I may offtracks but shouldn't we have a "check-test" step here too between "build" and "generated-build-stats". Or is this still pending because of works that still needs to be done with respect to `make check`? (indeed I don't see it in the default-actions configs either)

>diff --git a/taskcluster/ci/build/windows.yml b/taskcluster/ci/build/windows.yml
>--- a/taskcluster/ci/build/windows.yml
>+++ b/taskcluster/ci/build/windows.yml
>@@ -108,8 +108,29 @@ win64/pgo:
>+         using: mozharness
>+         script: "mozharness/scripts/fx_desktop_build.py"
>+         config:
>+             - builds/taskcluster_firefox_windows_64_addondevel.py 
>+             - balrog/production.py

Since its WIP, I think this is expected but we need actions set here for mozharness, right?
Ah, nevermind me, I see there's a default-actions list in the configs.

>diff --git a/taskcluster/taskgraph/transforms/gecko_v2_whitelist.py b/taskcluster/taskgraph/transforms/gecko_v2_whitelist.py
>--- a/taskcluster/taskgraph/transforms/gecko_v2_whitelist.py
>+++ b/taskcluster/taskgraph/transforms/gecko_v2_whitelist.py
>@@ -51,16 +51,18 @@ JOB_NAME_WHITELIST = set([
>+    'macosx64-st-an-opt',
>+    'macosx64-add-on-devel',

You might need to rebase, I see :dustin already added the `macosx64-st-an-opt` type here.

>diff --git a/testing/mozharness/configs/builds/taskcluster_firefox_windows_64_addondevel.py b/testing/mozharness/configs/builds/taskcluster_firefox_windows_64_addondevel.py
>new file mode 100644
>--- /dev/null
>+++ b/testing/mozharness/configs/builds/taskcluster_firefox_windows_64_addondevel.py
>@@ -0,0 +1,91 @@

>+    'base_name': 'browser/config/mozconfigs/win64/add-on-devel',

more like a heads-up, is this right? all the other configs have actual arch names here.

Comment 11

[Kim Moir [:kmoir] ET]

Attachment: addonsall.patch

Comment 12

[Kim Moir [:kmoir] ET]

Attachment: addonsall.patch

Comment 13

[Mihai Tabara [:mtabara]⌚️GMT]

Comment on attachment 8835004 [details] [diff] [review]
addonsall.patch

># HG changeset patch
># User Kim Moir <[e-mail]>
># Parent  86bc3b9a0943288f163fba65451c6f5482230a3e
>try: -b o -p linux64 -u none -t none
>diff --git a/testing/mozharness/configs/builds/taskcluster_firefox_windows_64_addondevel.py b/testing/mozharness/configs/builds/taskcluster_firefox_windows_64_addondevel.py
>new file mode 100644
>--- /dev/null
>+++ b/testing/mozharness/configs/builds/taskcluster_firefox_windows_64_addondevel.py
>@@ -0,0 +1,91 @@
>+import os
>+import sys
>+
>+config = {
>+    #########################################################################
>+    ######## WINDOWS GENERIC CONFIG KEYS/VAlUES
>+    # if you are updating this with custom 64 bit keys/values please add them
>+    # below under the '64 bit specific' code block otherwise, update in this
>+    # code block and also make sure this is synced between:
>+    # - taskcluster_firefox_win32_debug
>+    # - taskcluster_firefox_win32_opt
>+    # - taskcluster_firefox_win64_debug
>+    # - taskcluster_firefox_win64_opt
>+    # - taskcluster_firefox_win32_clang
>+    # - taskcluster_firefox_win32_clang_debug
>+    # - taskcluster_firefox_win64_clang
>+    # - taskcluster_firefox_win64_clang_debug
>+    # - taskcluster_firefox_win64_addonsdevel
>+
>+    'default_actions': [
>+        'clone-tools',
>+        'build',
>+        'check-test',
>+        'generate-build-stats'
>+    ],
>+    'exes': {
>+        'python2.7': sys.executable,
>+        'make': [
>+            sys.executable,
>+            os.path.join(
>+                os.getcwd(), 'build', 'src', 'build', 'pymake', 'make.py'
>+            )
>+        ],
>+        'virtualenv': [
>+            sys.executable,
>+            os.path.join(
>+                os.getcwd(), 'build', 'src', 'python', 'virtualenv', 'virtualenv.py'
>+            )
>+        ],
>+        'mach-build': [
>+            os.path.join(os.environ['MOZILLABUILD'], 'msys', 'bin', 'bash.exe'),
>+            os.path.join(os.getcwd(), 'build', 'src', 'mach'),
>+            '--log-no-times', 'build', '-v'
>+        ],
>+    },
>+    'app_ini_path': '%(obj_dir)s/dist/bin/application.ini',
>+    # decides whether we want to use moz_sign_cmd in env
>+    'enable_signing': False,
>+    'enable_ccache': False,
>+    'vcs_share_base': os.path.join('y:', os.sep, 'hg-shared'),
>+    'objdir': 'obj-firefox',
>+    'tooltool_script': [
>+      sys.executable,
>+      os.path.join(os.environ['MOZILLABUILD'], 'tooltool.py')
>+    ],
>+    'tooltool_bootstrap': 'setup.sh',
>+    'enable_count_ctors': False,
>+    'max_build_output_timeout': 60 * 80,
>+    #########################################################################
>+
>+
>+     #########################################################################
>+     ###### 64 bit specific ######
>+    'base_name': 'browser/config/mozconfigs/win64/add-on-devel',
Is this base_name right here? 

I see all other platforms have something like "WINNT_5.2_%(branch)s" or "WINNT_6.1_x86-64_%(branch)s"or "OS X 10.7 %(branch)s" or "Linux_%(branch)s" or "Android 2.3 %(branch)s". I'm just wondering if this shouldn't be something similar rather than a file path?

Comment 14

[Kim Moir [:kmoir] ET]

Attachment: addons.patch

fixed the base name and added win32 which I forgot earlier

Comment 15

[Kim Moir [:kmoir] ET]

Comment on attachment 8836209 [details] [diff] [review]
addons.patch

https://treeherder.mozilla.org/#/jobs?repo=try&revision=d5b6a3886294fe94358adf8fcdc9f388adbcec16&selectedJob=76442399

Comment 16

[Mihai Tabara [:mtabara]⌚️GMT]

Comment on attachment 8836209 [details] [diff] [review]
addons.patch

Looks good to me!

Quick question: are we sure we don't neet to set to True the `enable_signing` variable? I see it's turned on for most of the other platforms. The same with `publish_nightly_en_US_routes`.

Comment 17

[Kim Moir [:kmoir] ET]

Comment on attachment 8836209 [details] [diff] [review]
addons.patch

No, we don't want them signed.  They are also unbranded as specified in the mozconfigs.  This was one of the original requirements when they were implemented in buildbot as part of bug 1186522

Comment 18

[Mihai Tabara [:mtabara]⌚️GMT]

Comment on attachment 8836209 [details] [diff] [review]
addons.patch

Ok, thanks for clarifying that!
lgtm!

Comment 19

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/mozilla-inbound/rev/7f7725dd2c08
convert buildbot addon devel Linux64 builds to taskcluster r=mtabara

Comment 20

[Wes Kocher (:KWierso) (Not reading bugmail; email directly if needed)]

https://hg.mozilla.org/mozilla-central/rev/7f7725dd2c08