bugzilla.mozilla.org will be intermittently unavailable on Saturday, March 24th, from 16:00 until 20:00 UTC.

Add mozharness configuration options for running additional mach commands post-build and adding directories to the TaskCluster artifacts

RESOLVED FIXED in Firefox 47



Firefox for Android
Build Config & IDE Support
2 years ago
2 years ago


(Reporter: nalexander, Assigned: nalexander)


(Blocks: 1 bug)

Firefox 47

Firefox Tracking Flags

(firefox47 fixed)


MozReview Requests


Submitter Diff Changes Open Issues Last Updated
Error loading review requests:


(4 attachments, 1 obsolete attachment)



2 years ago
There are a number of Android Gradle jobs that look like:

1) run |mach build| with --disable-compile-environment
2) run |mach gradle COMMAND|
3) copy some directory from $OBJDIR/gradle/... into artifacts

jlund recommended adding a couple configuration switches to the long list for builds already present in the build mozharness script.

The first would be a list of mach commands |gradle COMMAND1|, |other-thing| to run after |mach build|.  (The whole job fails if either |mach build|, or any of the commands fails.)

The second would be a list of directories (in $OBJDIR) to upload as TC artifacts.  These need to be uploaded providing the |mach build| succeeds -- even when the additional commands fail.

That'll give us a general mechanism for doing these |mach build| and then ... style tasks.

Comment 1

2 years ago
jlund suggested the following place to put the flag:

diff --git a/testing/mozharness/configs/builds/releng_sub_android_configs/64_api_11.py b/testing/mozharness/configs/builds/releng_sub_android_configs/64_api_11_nicks_special_config.py
copy from testing/mozharness/configs/builds/releng_sub_android_configs/64_api_11.py
copy to testing/mozharness/configs/builds/releng_sub_android_configs/64_api_11_nicks_special_config.py
--- a/testing/mozharness/configs/builds/releng_sub_android_configs/64_api_11.py
+++ b/testing/mozharness/configs/builds/releng_sub_android_configs/64_api_11_nicks_special_config.py
@@ -1,8 +1,9 @@
 config = {
     'base_name': 'Android armv7 API 11+ %(branch)s',
     'stage_platform': 'android-api-11',
     'build_type': 'api-11-opt',
     'src_mozconfig': 'mobile/android/config/mozconfigs/android-api-11/nightly',
     'tooltool_manifest_src': 'mobile/android/config/tooltool-manifests/android/releng.manifest',
     'multi_locale_config_platform': 'android',
+    'nicks_awesome_post_build_thing': True,
diff --git a/testing/mozharness/mozharness/mozilla/building/buildbase.py b/testing/mozharness/mozharness/mozilla/building/buildbase.py
--- a/testing/mozharness/mozharness/mozilla/building/buildbase.py
+++ b/testing/mozharness/mozharness/mozilla/building/buildbase.py
@@ -1863,18 +1863,21 @@ or run without that action (ie: --no-{ac
     def postflight_build(self, console_output=True):
         """grabs properties from post build and calls ccache -s"""
         if self.config.get('enable_ccache'):
+        if self.config.get("nicks_awesome_post_build_thing"):
+            self.nick_has_the_power()  # this will get run after we build fennec
     def preflight_package_source(self):
     def package_source(self):
         """generates source archives and uploads them"""
         env = self.query_build_env()
         python = self.query_exe('python2.7')

Comment 2

2 years ago
Created attachment 8722883 [details]
MozReview Request: Bug 1249166 - Pre: Set JAVA_HOME in |mach gradle|. r=me

Review commit: https://reviewboard.mozilla.org/r/36275/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/36275/

Comment 3

2 years ago
Created attachment 8722884 [details]
MozReview Request: Bug 1249166 - Pre: Fix Android front-end TreeHerder details; force --no-daemon and --offline in automation. r=me

Setting GRADLE_USER_HOME in this way ensures all Gradle invocations in
automation have the right flags, rather than just the ones we

Review commit: https://reviewboard.mozilla.org/r/36277/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/36277/

Comment 4

2 years ago
Created attachment 8722885 [details]
MozReview Request: Bug 1249166 - Part 1: Add post-build mach commands to mozharness. r?jlund

This allows to run |mach gradle COMMAND| after the initial |mach build|.

Review commit: https://reviewboard.mozilla.org/r/36279/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/36279/
Attachment #8722885 - Flags: review?(jlund)

Comment 5

2 years ago
Created attachment 8722886 [details]
MozReview Request: Bug 1249166 - Part 2: Copy Gradle artifacts. r?dustin

I originally implemented this as a fairly generic extension to
mozharness that allowed to copy additional files from the object
directory.  There are two problems with this approach.  First, my
tasks disable uploading files (since we're not really packaging --
this is only *like* a build), so we can't use the buildbot-compatible
TaskCluster upload routines anyway.  Second, in this case, one might
just copy the files into the artifacts directory -- but buildbot
really shouldn't know the path to TaskCluster directly.  So here we
are, an expedient way to bridge buildbot and TaskCluster.

Review commit: https://reviewboard.mozilla.org/r/36281/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/36281/
Attachment #8722886 - Flags: review?(dustin)

Comment 6

2 years ago

::: testing/taskcluster/tasks/builds/android_api_15_frontend.yml:46
(Diff revision 1)
> -    command: ["/bin/bash", "bin/build.sh"]
> +    command: 

I'll kill the whitespace before landing.
Comment on attachment 8722886 [details]
MozReview Request: Bug 1249166 - Part 2: Copy Gradle artifacts. r?dustin


::: testing/taskcluster/tasks/builds/android_api_15_frontend.yml:54
(Diff revision 1)
> +        true

The task definition isn't the right place to write a shell script.

I don't understand in detail your reasons for not doing this in Mozharness, but I have had similar difficulties threading things through the maze, so I understand in principle.

So, there are a few other options here.

1. Add these locations as additional artifact directories, so that the docker-worker uploads them for you (no need to copy them first).  This is probably the cleanest.

2. Modify build-linux.sh to perform this copy after the Mozharness run, regardless of its success, preferably in a more generic way.  Perhaps by setting POST_RUN_ARTIFACT_COPIES to a space-separated list of source, destination directories?

3. `build/upload.py` already supports uploading to an artifact directory, so you could just enable uploading and configure these directories appropriately:
Attachment #8722886 - Flags: review?(dustin)

Comment 8

2 years ago
Created attachment 8723116 [details]
MozReview Request: Bug 1249166 - Part 2: Copy Gradle artifacts. r?dustin

Review commit: https://reviewboard.mozilla.org/r/36361/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/36361/
Attachment #8723116 - Flags: review?(dustin)


2 years ago
Attachment #8722886 - Attachment is obsolete: true

Comment 9

2 years ago
Comment on attachment 8722885 [details]
MozReview Request: Bug 1249166 - Part 1: Add post-build mach commands to mozharness. r?jlund


if it works for you in testing/try, it looks good to me. :)

::: testing/mozharness/mozharness/mozilla/building/buildbase.py:1884
(Diff revision 1)
> +        # A list of argument lists.  Better names gratefully accepted!

names seem fine to me! I really like this extendable approach..
Attachment #8722885 - Flags: review?(jlund) → review+
Comment on attachment 8723116 [details]
MozReview Request: Bug 1249166 - Part 2: Copy Gradle artifacts. r?dustin

Attachment #8723116 - Flags: review?(dustin) → review+

Comment 12

2 years ago
Last Resolved: 2 years ago
status-firefox47: --- → fixed
Resolution: --- → FIXED
Target Milestone: --- → Firefox 47
Assignee: nobody → nalexander
You need to log in before you can comment on or make changes to this bug.