1055918 - switch all remaining desktop builders on trunk branches to use mozharness mach

Status: RESOLVED FIXED

(Release Engineering :: General, defect)

Description

[Jordan Lund (:jlund)]

this should include try

Comment 1

[[e-mail] (:bhearsum)]

I suspect you probably know this already, but if at all possible, mozilla-central, inbounds (m-i, b2g-i, etc.), and try should all be changed in the same reconfig - it helps reduce issues where someone tests on try and then busts a tree when they land.

Comment 2

[Jordan Lund (:jlund)]

(In reply to Ben Hearsum [:bhearsum] from comment #1)
> I suspect you probably know this already, but if at all possible,
> mozilla-central, inbounds (m-i, b2g-i, etc.), and try should all be changed
> in the same reconfig - it helps reduce issues where someone tests on try and
> then busts a tree when they land.

yup, thanks! I should have made it clear in description that was my intention. appreciate the feedback/tip. :)

Comment 3

[Jordan Lund (:jlund)]

Attachment: 2014_1055918_mach_mozharness_trunk.fatal_if_mach_build_fails.patch

this mainly is about self.fatal'ing if mach build fails.

other is minor pep fix and fighting bitrot: mozilla-b2g32_v2_0 branch and removing UPLOAD_HOST from try env as it is now referenced only in build_pool_specifics.py

Comment 4

[Jordan Lund (:jlund)]

Attachment: 2014_1055918_mach_mozharness_trunk.fatal_if_mach_build_fails2.patch

Comment 5

[Jordan Lund (:jlund)]

Attachment: 2014_1055918_mach_mozharness_trunk.fatal_if_mach_build_fails3.patch

might as well make it 3 patches :)

this one allows for 'mach build' to return 1 as a valid return code, else force it to 2.

Comment 6

[Jordan Lund (:jlund)]

Attachment: 20141015_1055918_enable_twigs_mozharness_mach.bbotcfgs.patch

rail,

I'm going to enable this on trunk in two stages.

1) twigs 2) trunk

The main reason is I've been clobbering branches to clean up new file structure of the builddir (things look different with mozharness vs MozillaBuildFactory builds). Also so I can keep testing various variants in production that have yet to be tested. e.g. pgo builds are not done on cypress,cedar,ash so even though I have done nightlies(use pgo to compile), there is subtle differences with pgo itself and our automation steps.

I can give you a dumpmaster but it will be hard to grep as I am changing the entire builder for these. Here is the builderdiff: http://people.mozilla.org/~jlund/mach_mozharness_builderdiff.diff

Comment 7

[Rail Aliiev [:rail]]

Comment on attachment 8505566 [details] [diff] [review]
20141015_1055918_enable_twigs_mozharness_mach.bbotcfgs.patch

ship it!

Comment 8

[Jordan Lund (:jlund)]

Comment on attachment 8505566 [details] [diff] [review]
20141015_1055918_enable_twigs_mozharness_mach.bbotcfgs.patch

thanks, this has landed and is in production

Comment 9

[Chris Cooper [:coop] (he/him)]

Merged to production, and deployed.

Comment 10

[Jordan Lund (:jlund)]

I got to test out nightlies manually on fx-team. it turns out that we are not uploading the partial mar we create so updating is failing and we have to download a whole new nightly.

Digging through the code, it looks like after we moved targets into mach, e.g. the target to generate complete mar and to upload it, we left partial mar generation outside of mach. Obviously this does not work as it results in mozharness creating the partial after 'mach build' calls 'make upload'[1] Doh!

we left partial mars out as the code is a bit long/messy.

this means we have two options:

1) rip out make upload from within mach build. and then add 'make upload' and the parser back into mozharness[2]. this will result in mach barely setting anything in mach_build_properties.json

2) add partial generation logic into mach build[3]

[1] http://mxr.mozilla.org/build/source/mozharness/mozharness/mozilla/building/buildbase.py#1556
[2] http://hg.mozilla.org/build/mozharness/file/cc13e75a4318/mozharness/mozilla/building/buildbase.py#l1471
[3] http://mxr.mozilla.org/build/source/mozharness/mozharness/mozilla/building/buildbase.py#1241

mshal: either option, I'll need to coordinate with you 'one more time'. ;)

Comment 11

[Jordan Lund (:jlund)]

it may be worth inquiring about funsize, the tool that generates partials:

17:05:38 <hwine> jlund: mshal fyi - some of the work on funsize might also
simplify that code. The bash tooling is being moved to python
17:06:24 <jlund> hwine: is funsize ready to use? can we make partials from a
complete mar in production?
17:07:10 <hwine> jlund: no, not in production, but thinking there's syncergy
there if any work needs doing
17:07:41 <hwine> (we wouldn't want devs to have to have network access to
funsize to test updaters anyway)
17:09:40 <jlund> hwine: sorry, bear with me here, how can we use funsize
instead of:
http://mxr.mozilla.org/build/source/mozharness/mozharness/mozilla/building/buildbase.py#1241
? for trunk
17:12:10 <hwine> jlund: I phrased wrong. You can not use funsize. However,
funsize needs to change the code invoked there from a perl script to python.
That _may_ simplify things inside mach, and make it worthwhile to do first
17:12:40 <hwine> there == buildbase.py#1241

Comment 12

[Michael Shal [:mshal]]

(In reply to Jordan Lund (:jlund) from comment #10)
> 1) rip out make upload from within mach build. and then add 'make upload'
> and the parser back into mozharness[2]. this will result in mach barely
> setting anything in mach_build_properties.json

This should be easy to do, but has implications with bug 1084163 that glandium filed. Specifically, how would mozharness know to run 'make upload' if the 'make check' part of the build fails? We'd have the same issue we have with deciding to turn the build orange - ie: we probably need to parse the build log or come up with some other hack in order to differentiate check failures. Or would we just always run 'make upload' even if the build didn't complete at all? (And possibly upload stale/broken build artifacts).

> 2) add partial generation logic into mach build[3]
> 
> [1]
> http://mxr.mozilla.org/build/source/mozharness/mozharness/mozilla/building/
> buildbase.py#1556
> [2]
> http://hg.mozilla.org/build/mozharness/file/cc13e75a4318/mozharness/mozilla/
> building/buildbase.py#l1471
> [3]
> http://mxr.mozilla.org/build/source/mozharness/mozharness/mozilla/building/
> buildbase.py#1241
> 
> mshal: either option, I'll need to coordinate with you 'one more time'. ;)

I had hoped to avoid this due to the fact that funsize was on the horizon. However, I may have misunderstood the goals/features of what funsize would offer. I thought it would be generating partial mars on-the-fly, so we wouldn't be generating them in the build at all (ie: neither through the buildbot hackery, nor as part of 'mach build'). Meaning if funsize was in production, we would be ripping out the buildbot pieces to do partial mar generation. Am I wrong there?

I'm a little leery of going down the road of trying to move all of partial mar generation into mach build due to the complexity involved, if it means that by the time we finish it, funsize is ready and it's time to remove all that code anyway.

But if funsize means that we'll still have something in the build process, just less complicated than what it is now, then I guess moving it into mach build is the way to go. Is hwine the best person to chat with about this? Maybe the 3 of us should get together & discuss.

Comment 13

[Michael Shal [:mshal]]

glandium, thanks for filing these followups!

Comment 14

[Phil Ringnalda (:philor)]

(In reply to Jordan Lund (:jlund) from comment #2)
> (In reply to Ben Hearsum [:bhearsum] from comment #1)
> > I suspect you probably know this already, but if at all possible,
> > mozilla-central, inbounds (m-i, b2g-i, etc.), and try should all be changed
> > in the same reconfig - it helps reduce issues where someone tests on try and
> > then busts a tree when they land.
> 
> yup, thanks! I should have made it clear in description that was my
> intention. appreciate the feedback/tip. :)

The "etc." in that list includes the inbound named fx-team, where you did enable it separately.

For bonus fun, the Jetpack tree uses a simplistic method to find the most recent build on fx-team, looking for the most recent directory on ftp.m.o and assuming there will be a build in it, and mozharness builds seem to be uploading things to utterly random directories (http://ftp.mozilla.org/pub/mozilla.org/firefox/tinderbox-builds/fx-team-macosx64-debug/1413663134/ contains nothing but a log saying its build is in http://ftp.mozilla.org/pub/mozilla.org/firefox/tinderbox-builds/fx-team-macosx64-debug/1413569808/, and that directory not only has that build, but also the build log from a build from the day before, and the test logs from the day before, so I think it just uploaded over the top of a previous build on a different rev, and non-unified builds which used to upload their log-only to -nonunified are now dumping it in the same directory with the unified builds), breaking that simplistic method.

Comment 15

[Phil Ringnalda (:philor)]

Filed that separately as bug 1085055 since I needed a place to be the explanation for the fx-team tree closure.

Comment 16

[Nick Thomas [:nthomas] (UTC+12)]

In bug 1085055 I added fx-team to the list of branches which don't use mach mozharness yet.

Comment 17

[Erik Vold [:erikvold] (please needinfo? me)]

(In reply to Phil Ringnalda (:philor) from comment #14) 
> For bonus fun, the Jetpack tree uses a simplistic method to find the most
> recent build on fx-team, looking for the most recent directory on ftp.m.o
> and assuming there will be a build in it, and mozharness builds seem to be
> uploading things to utterly random directories
> (http://ftp.mozilla.org/pub/mozilla.org/firefox/tinderbox-builds/fx-team-
> macosx64-debug/1413663134/ contains nothing but a log saying its build is in
> http://ftp.mozilla.org/pub/mozilla.org/firefox/tinderbox-builds/fx-team-
> macosx64-debug/1413569808/, and that directory not only has that build, but
> also the build log from a build from the day before, and the test logs from
> the day before, so I think it just uploaded over the top of a previous build
> on a different rev, and non-unified builds which used to upload their
> log-only to -nonunified are now dumping it in the same directory with the
> unified builds), breaking that simplistic method.

Can the "latest" directory in http://ftp.mozilla.org/pub/mozilla.org/firefox/tinderbox-builds/fx-team-macosx64-debug/ be used instead of this largest number lookup stuff https://github.com/mozilla/build-tools/blob/master/buildfarm/utils/run_jetpack.py#L156-L173 ?

Comment 18

[Phil Ringnalda (:philor)]

You wanted either bug 1085053 or perhaps bug 758419 for that, but the twin answers are "directory dates aren't a source of truth" and "you want to switch yourself (because nobody else will do it for you) to using https://github.com/mozilla/mozdownload instead of anything hand-rolled in ten minutes."

Comment 19

[Jordan Lund (:jlund)]

Attachment: 141027_bug_1055918_fixes_for_mh_mach_twig_branches.patch

tested on: https://tbpl.mozilla.org/?tree=Ash&rev=2bee4e7591f9

list of fixes outlined in next comment

Comment 20

[Jordan Lund (:jlund)]

Comment on attachment 8512158 [details] [diff] [review]
141027_bug_1055918_fixes_for_mh_mach_twig_branches.patch

Review of attachment 8512158 [details] [diff] [review]:
-----------------------------------------------------------------

::: configs/builds/branch_specifics.py
@@ -71,5 @@
>      'mozilla-b2g30_v1_4': {
>          'repo_path': 'releases/mozilla-b2g30_v1_4',
>          'use_branch_in_symbols_extra_buildid': False,
>          'update_channel': 'nightly-b2g30',
> -        'branch_supports_partials': False,

partials are all done in mach now, we don't worry about them at a mozharness level

@@ -94,5 @@
> -            # stage_server is dictated from build_pool_specifics.py
> -            'UPLOAD_USER': "trybld",
> -            'UPLOAD_TO_TEMP': '1',
> -            'UPLOAD_SSH_KEY': '~/.ssh/%s' % ("trybld_dsa",),
> -        },

for upload_env, we use staging_server, staging_user, and staging_ssh_key that is defined at build_pool_specifics.py or overrided in this ['try'] {} dict.

::: configs/builds/releng_base_linux_32_builds.py
@@ -16,5 @@
>          'clobber',
>          'clone-tools',
>          'setup-mock',
>          'build',
> -        'sendchanges',

sendchanges is no longer an action, It is called as a post step to build

@@ +42,5 @@
>          ('/builds/gapi.data', '/builds/gapi.data'),
>          ('/tools/tooltool.py', '/builds/tooltool.py'),
>      ],
>      'enable_ccache': True,
> +    'enable_check_test': True,

we have re-added check test into mozharness from mach so we need to know when to run it and not (e.g. it does not run for non-unified builds)

@@ -148,5 @@
>      ],
>      'src_mozconfig': 'browser/config/mozconfigs/linux32/nightly',
>      'tooltool_manifest_src': "browser/config/tooltool-manifests/linux32/\
>  releng.manifest",
> -    'platform_ftp_name': 'linux-i686.complete.mar',

this was part of the partial logic, which again, mach handles

::: configs/builds/releng_base_windows_32_builds.py
@@ +94,5 @@
>      "check_test_env": {
>          'MINIDUMP_STACKWALK': '%(abs_tools_dir)s/breakpad/win32/minidump_stackwalk.exe',
>          'MINIDUMP_SAVE_PATH': '%(base_work_dir)s/minidumps',
>      },
> +    'enable_pymake': True,

since we are doing a make cmd, we need pymake for windows

::: configs/builds/releng_sub_linux_configs/64_stat_and_debug.py
@@ +18,5 @@
>  clang.manifest",
>      'platform_supports_post_upload_to_latest': False,
>      'enable_signing': False,
> +    'enable_talos_sendchange': False,
> +    'enable_unittest_sendchange': False,

since we removed the sendchange action, we need these keys in every platform config to say whether we are doing talos or unittest sendchanges. they were missing here since we previously had the sendchange action (now gone) commented out

::: mozharness/mozilla/building/buildbase.py
@@ +75,5 @@
>          'level': TBPL_RETRY,
>      }
>  ]
>  
> +class CheckTestCompleteParser(OutputParser):

used to check the output of 'make -k check'

@@ -89,5 @@
> -        ('robocopApkUrl', "m.endswith('apk') and 'robocop' in m"),
> -        ('jsshellUrl', "'jsshell-' in m and m.endswith('.zip')"),
> -        ('partialMarUrl', "m.endswith('.mar') and '.partial.' in m"),
> -        ('completeMarUrl', "m.endswith('.mar')"),
> -    ]

whoops, I just double checked and it seems that MakeUploadOutputParser is used in http://mxr.mozilla.org/build/source/mozharness/scripts/b2g_build.py#35

I'll re-add that now in a follow up patch (incoming now)

@@ +703,5 @@
>                  env["MOZ_UPDATE_CHANNEL"] = c['update_channel']
>              else:  # let's just give the generic channel based on branch
>                  env["MOZ_UPDATE_CHANNEL"] = "nightly-%s" % (self.branch,)
>  
> +        if self.config.get('pgo_build') or self._compile_against_pgo():

_compile_against_pgo is to fix https://bugzilla.mozilla.org/show_bug.cgi?id=1087100

see bug for more details

@@ +731,5 @@
> +                'stage_username': c['stage_username']
> +            }
> +            mach_env['UPLOAD_SSH_KEY'] = mach_env['UPLOAD_SSH_KEY'] % {
> +                'stage_ssh_key': c['stage_ssh_key']
> +            }

this are the script changes that corresponds to all the UPLOAD_* config changes

@@ +736,5 @@
> +
> +        if self.query_is_nightly():
> +            mach_env['LATEST_MAR_DIR'] = c['latest_mar_dir'] % {
> +                'branch': self.branch
> +            }

latest_mar_dir is difficult for mach to get so we pass it via ENV for partial's sake

@@ +834,5 @@
>          else:
>              # the default
>              tinderbox_build_dir = "%s-%s" % (self.branch, platform)
>  
> +        if who and self.branch == 'try':

we don't pass who to try branches. this was a bug

@@ +847,5 @@
>          if revision:
>              post_upload_cmd.extend(['--revision', revision])
>          if c.get('to_tinderbox_dated'):
>              post_upload_cmd.append('--release-to-tinderbox-dated-builds')
> +            post_upload_cmd.append('--release-to-latest-tinderbox-builds')

this was also missing in post_upload.py step.

@@ +863,5 @@
>          dirs = self.query_abs_dirs()
>          env = self.query_build_env()
> +        self.run_command(command=['ccache', '-z'],
> +                         cwd=dirs['base_work_dir'],
> +                         env=env)

we don't care if src is there or not since ccache details are stored in ~/

@@ +1315,5 @@
>                        os.path.join(dirs['abs_work_dir'], 'buildprops.json'))
>  
>          python = self.query_exe('python2.7')
>          return_code = self.run_command_m(
> +            command=[python, 'mach', '--log-no-times', 'build', '-v'],

bug fixes for:

bug 1084170, bug 1084165

@@ +1327,3 @@
>              )
> +            self.fatal("'mach build' did not run successfully. Please check "
> +                       "log for errors.")

we only need to worry if mach build returns a non 0. if that's the case, we should fatal

Comment 21

[Jordan Lund (:jlund)]

Attachment: 141023_bug_1055918_trunk_branches_mozharness_mach-include-makeupload-parser.patch

interdiff from last patch. as mentioned in last comment, adds the upload parser again as it's used outside of desktop builds

Comment 22

[Massimo Gervasini [:massimo]]

Hi Jordan,

your patches look fine to me but I haven't finished my testing because the first one does not apply on a clean mozharness checkout:

hg import https://bug1055918.bugzilla.mozilla.org/attachment.cgi?id=8512158
patching file mozharness/mozilla/building/buildbase.py
Hunk #1 FAILED at 12
1 out of 6 hunks FAILED -- saving rejects to file mozharness/mozilla/building/buildbase.py.rej
abort: patch failed to apply

Comment 23

[Jordan Lund (:jlund)]

Attachment: 141028_bug_1055918_fixes_for_mh_mach_twig_branches.patch

sorry, this patch was split from other bugs. those bugs themselves had follow up patches that landed on mainline mozharness causing this to no longer apply

Comment 24

[Massimo Gervasini [:massimo]]

Comment on attachment 8512807 [details] [diff] [review]
141028_bug_1055918_fixes_for_mh_mach_twig_branches.patch

Hi Jordan,

this patch looks overall good to me, I have just one question, before r+:


in mozharness/mozilla/building/buildbase.py

>             self.return_code = self.worst_level(
>-                return_code,  self.return_code, AUTOMATION_EXIT_CODES[::-1]
>+                EXIT_STATUS_DICT[TBPL_FAILURE],  self.return_code,
>+                AUTOMATION_EXIT_CODES[::-1]

TBPL_FAILURE is used but not defined, I am going to assume the import statement got lost during the generation of this patch (it was imported in the obsolete one). 
Can you confirm that my assumption is correct? If not, there might something strange going on with the return code. 

other (minor) changes:
in configs/builds/releng_base_linux_32_builds.py and configs/builds/releng_base_linux_64_builds.py
can we remove 'import sys' (unused) ?

Comment 25

[Jordan Lund (:jlund)]

thanks Massimo :D

> 
> in mozharness/mozilla/building/buildbase.py
> 
> >             self.return_code = self.worst_level(
> >-                return_code,  self.return_code, AUTOMATION_EXIT_CODES[::-1]
> >+                EXIT_STATUS_DICT[TBPL_FAILURE],  self.return_code,
> >+                AUTOMATION_EXIT_CODES[::-1]
> 
> TBPL_FAILURE is used but not defined, I am going to assume the import
> statement got lost during the generation of this patch (it was imported in
> the obsolete one). 
> Can you confirm that my assumption is correct? If not, there might something
> strange going on with the return code.

yes that was lost in translation!

> 
> other (minor) changes:
> in configs/builds/releng_base_linux_32_builds.py and
> configs/builds/releng_base_linux_64_builds.py
> can we remove 'import sys' (unused) ?

good catch.


interdiff:


diff --git a/configs/builds/releng_base_linux_32_builds.py b/configs/builds/releng_base_linux_32_builds.py
index 4684ad4..be4d67b 100644
--- a/configs/builds/releng_base_linux_32_builds.py
+++ b/configs/builds/releng_base_linux_32_builds.py
@@ -1,5 +1,4 @@
 import os
-import sys

 STAGE_USERNAME = 'ffxbld'
 STAGE_SSH_KEY = 'ffxbld_rsa'
diff --git a/configs/builds/releng_base_linux_64_builds.py b/configs/builds/releng_base_linux_64_builds.py
index 750dd29..80aa7df 100644
--- a/configs/builds/releng_base_linux_64_builds.py
+++ b/configs/builds/releng_base_linux_64_builds.py
@@ -1,5 +1,4 @@
 import os
-import sys

 STAGE_USERNAME = 'ffxbld'
 STAGE_SSH_KEY = 'ffxbld_rsa'
diff --git a/mozharness/mozilla/building/buildbase.py b/mozharness/mozilla/building/buildbase.py
index b9f482b..ead1387 100755
--- a/mozharness/mozilla/building/buildbase.py
+++ b/mozharness/mozilla/building/buildbase.py
@@ -31,7 +31,7 @@ from mozharness.base.script import PostScriptRun
 from mozharness.base.vcs.vcsbase import MercurialScript
 from mozharness.mozilla.buildbot import BuildbotMixin, TBPL_STATUS_DICT, \
     TBPL_EXCEPTION, TBPL_RETRY, EXIT_STATUS_DICT, TBPL_WARNING, TBPL_SUCCESS, \
-    TBPL_WORST_LEVEL_TUPLE
+    TBPL_WORST_LEVEL_TUPLE, TBPL_FAILURE
 from mozharness.mozilla.purge import PurgeMixin
 from mozharness.mozilla.mock import MockMixin
 from mozharness.mozilla.signing import SigningMixin

Comment 26

[Massimo Gervasini [:massimo]]

Comment on attachment 8512807 [details] [diff] [review]
141028_bug_1055918_fixes_for_mh_mach_twig_branches.patch

r+ with changes in comment #25

Comment 27

[Jordan Lund (:jlund)]

Comment on attachment 8512807 [details] [diff] [review]
141028_bug_1055918_fixes_for_mh_mach_twig_branches.patch

https://hg.mozilla.org/build/mozharness/rev/42d035f84c63

Comment 28

[Chris Cooper [:coop] (he/him)]

In production: https://hg.mozilla.org/build/mozharness/rev/42d035f84c63

Comment 29

[Jordan Lund (:jlund)]

Attachment: 20141030_1055918_mozharness_mach_nonunifed_uploadsymbol_fix.mozharness.patch

this:

- stops env vars from being passed to mach build for try builds (we don't uploadsymbols for those builds anyway)
- we don't want to set a buildbot prop if mach set it to "UNKNOWN"
- postrun.py needs a stage_platform buildbot property to set the right log url. It falls back to platform prop but this breaks things for builds like nonunified that have more specific stage_platform values. mozharness didn't know stage_platform was swallowed downstream in buildbot world.

ash push (waiting on result): https://tbpl.mozilla.org/?tree=Ash&rev=17b74b3727ef

Comment 30

[Massimo Gervasini [:massimo]]

Comment on attachment 8514810 [details] [diff] [review]
20141030_1055918_mozharness_mach_nonunifed_uploadsymbol_fix.mozharness.patch

Looks good to me!

Comment 31

[Jordan Lund (:jlund)]

Comment on attachment 8514810 [details] [diff] [review]
20141030_1055918_mozharness_mach_nonunifed_uploadsymbol_fix.mozharness.patch

on default: http://hg.mozilla.org/build/mozharness/rev/8271db6faea5

Comment 32

[Chris Cooper [:coop] (he/him)]

In production: https://hg.mozilla.org/build/mozharness/rev/8271db6faea5

Comment 33

[Jordan Lund (:jlund)]

Attachment: nightly update failing on maple

tesing nightlies on maple, I manually downloaded a dmg from: http://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/2014/10/2014-10-31-04-02-01-maple/

After running and checking for updates, it fails to update with the above screen shot.

I believe it tries to install nightly from the latest: https://aus4-admin.mozilla.org/releases/Firefox-maple-nightly-20141101040202/data

There are aus rules in place for maple nightlies so that is not the issue. 

nthomas suggested we need to do something like: https://hg.mozilla.org/projects/alder/rev/a752028423fd for maple.

I am going to try to get this working in ash so we can continue to iterate in a testing env without breaking other people's project branches.

so for ash, since we already have nightlies, IIUC we only need to add a aus rule (https://bugzilla.mozilla.org/show_bug.cgi?id=1083853#c10) and then add nightly-ash to https://hg.mozilla.org/projects/alder/rev/a752028423fd

Comment 34

[Jordan Lund (:jlund)]

> tesing nightlies on maple, I manually downloaded a dmg from:
> http://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/2014/10/2014-10-31-04-
> 02-01-maple/
> 
> After running and checking for updates, it fails to update with the above
> screen shot.
> 

It looks like maple hasn't picked up any recent m-c changes so bug 1087104 and 1085026 have not landed there yet causing nightlies to not apply.

I added a balrog rule to ash so the nightly_channel would be picked up. After manually applying an older version of ff, applying the update worked and applied. \o/ win.

https://bugzilla.mozilla.org/attachment.cgi?id=8512807 resolves 1084165, 1084170, 1087100, and 1087104

we are now ready to try landing this on remaining trunk branches. patch to flip those branches incoming

Comment 35

[Jordan Lund (:jlund)]

Attachment: 141105_bug_1055918_stage_platform_revert_branch_upload_check_and_orange_if_check_fails.patch

this:

* reverts branch_supports_uploadsymbols from previous patch (this stops upload env vars which is bad as we wanted to stop uploadsymbol vars). The actual fix is not needed anyway since we do not run nightlies on try anyway (uploadsymbols only runs on nightlies)

* Bug 1093911 - when make check fails with mozharness, the build color does not change

* fixes https://bugzilla.mozilla.org/show_bug.cgi?id=1087101#c12

tested on staging master

Comment 36

[Massimo Gervasini [:massimo]]

Comment on attachment 8517571 [details] [diff] [review]
141105_bug_1055918_stage_platform_revert_branch_upload_check_and_orange_if_check_fails.patch

lgtm

Comment 37

[Jordan Lund (:jlund)]

Comment on attachment 8517571 [details] [diff] [review]
141105_bug_1055918_stage_platform_revert_branch_upload_check_and_orange_if_check_fails.patch

on default: https://hg.mozilla.org/build/mozharness/rev/1c6f07983bb2

Comment 38

[Pete Moore [:pmoore][:pete]]

Checked in code deployed to production

Comment 39

[Jordan Lund (:jlund)]

Attachment: 141110_bug_1055918_mozharness_mach_on_trunk.patch

Comment 40

[[e-mail] (:bhearsum)]

Comment on attachment 8520164 [details] [diff] [review]
141110_bug_1055918_mozharness_mach_on_trunk.patch

Review of attachment 8520164 [details] [diff] [review]:
-----------------------------------------------------------------

::: mozilla/config.py
@@ -2881,5 @@
>  # enable mozharness desktop builds across all twigs
>  for name, branch in items_at_least(BRANCHES, 'gecko_version', mc_gecko_version):
> -    if name in ('mozilla-central', 'mozilla-inbound', 'b2g-inbound', 'fx-team', 'try'):
> -        # only enable on twigs for now
> -        continue

Watch out for fx-team having nightlies too - you may want to lock their rule too.

The comment on line 2881 needs to be updated too.

Comment 41

[Jordan Lund (:jlund)]

> 
> Watch out for fx-team having nightlies too - you may want to lock their rule
> too.

IIUC fx-team doesn't do nightlies. I think UX does but they haven't since may.

> 
> The comment on line 2881 needs to be updated too.

comment fixed

Comment 42

[Jordan Lund (:jlund)]

Comment on attachment 8520164 [details] [diff] [review]
141110_bug_1055918_mozharness_mach_on_trunk.patch

on default: https://hg.mozilla.org/build/buildbot-configs/rev/5a9b98575bc9

Comment 43

[Jordan Lund (:jlund)]

Comment on attachment 8520164 [details] [diff] [review]
141110_bug_1055918_mozharness_mach_on_trunk.patch

in production