Closed Bug 1249166 Opened 6 years ago Closed 6 years ago

Add mozharness configuration options for running additional mach commands post-build and adding directories to the TaskCluster artifacts


(Firefox Build System :: Android Studio and Gradle Integration, defect)

Not set


(firefox47 fixed)

Tracking Status
firefox47 --- fixed


(Reporter: nalexander, Assigned: nalexander)




(4 files, 1 obsolete file)

There are a number of Android Gradle jobs that look like:

1) run |mach build| with --disable-compile-environment
2) run |mach gradle COMMAND|
3) copy some directory from $OBJDIR/gradle/... into artifacts

jlund recommended adding a couple configuration switches to the long list for builds already present in the build mozharness script.

The first would be a list of mach commands |gradle COMMAND1|, |other-thing| to run after |mach build|.  (The whole job fails if either |mach build|, or any of the commands fails.)

The second would be a list of directories (in $OBJDIR) to upload as TC artifacts.  These need to be uploaded providing the |mach build| succeeds -- even when the additional commands fail.

That'll give us a general mechanism for doing these |mach build| and then ... style tasks.
jlund suggested the following place to put the flag:

diff --git a/testing/mozharness/configs/builds/releng_sub_android_configs/ b/testing/mozharness/configs/builds/releng_sub_android_configs/
copy from testing/mozharness/configs/builds/releng_sub_android_configs/
copy to testing/mozharness/configs/builds/releng_sub_android_configs/
--- a/testing/mozharness/configs/builds/releng_sub_android_configs/
+++ b/testing/mozharness/configs/builds/releng_sub_android_configs/
@@ -1,8 +1,9 @@
 config = {
     'base_name': 'Android armv7 API 11+ %(branch)s',
     'stage_platform': 'android-api-11',
     'build_type': 'api-11-opt',
     'src_mozconfig': 'mobile/android/config/mozconfigs/android-api-11/nightly',
     'tooltool_manifest_src': 'mobile/android/config/tooltool-manifests/android/releng.manifest',
     'multi_locale_config_platform': 'android',
+    'nicks_awesome_post_build_thing': True,
diff --git a/testing/mozharness/mozharness/mozilla/building/ b/testing/mozharness/mozharness/mozilla/building/
--- a/testing/mozharness/mozharness/mozilla/building/
+++ b/testing/mozharness/mozharness/mozilla/building/
@@ -1863,18 +1863,21 @@ or run without that action (ie: --no-{ac
     def postflight_build(self, console_output=True):
         """grabs properties from post build and calls ccache -s"""
         if self.config.get('enable_ccache'):
+        if self.config.get("nicks_awesome_post_build_thing"):
+            self.nick_has_the_power()  # this will get run after we build fennec
     def preflight_package_source(self):
     def package_source(self):
         """generates source archives and uploads them"""
         env = self.query_build_env()
         python = self.query_exe('python2.7')
Setting GRADLE_USER_HOME in this way ensures all Gradle invocations in
automation have the right flags, rather than just the ones we

Review commit:
See other reviews:
This allows to run |mach gradle COMMAND| after the initial |mach build|.

Review commit:
See other reviews:
Attachment #8722885 - Flags: review?(jlund)
I originally implemented this as a fairly generic extension to
mozharness that allowed to copy additional files from the object
directory.  There are two problems with this approach.  First, my
tasks disable uploading files (since we're not really packaging --
this is only *like* a build), so we can't use the buildbot-compatible
TaskCluster upload routines anyway.  Second, in this case, one might
just copy the files into the artifacts directory -- but buildbot
really shouldn't know the path to TaskCluster directly.  So here we
are, an expedient way to bridge buildbot and TaskCluster.

Review commit:
See other reviews:
Attachment #8722886 - Flags: review?(dustin)

::: testing/taskcluster/tasks/builds/android_api_15_frontend.yml:46
(Diff revision 1)
> -    command: ["/bin/bash", "bin/"]
> +    command: 

I'll kill the whitespace before landing.
Comment on attachment 8722886 [details]
MozReview Request: Bug 1249166 - Part 2: Copy Gradle artifacts. r?dustin

::: testing/taskcluster/tasks/builds/android_api_15_frontend.yml:54
(Diff revision 1)
> +        true

The task definition isn't the right place to write a shell script.

I don't understand in detail your reasons for not doing this in Mozharness, but I have had similar difficulties threading things through the maze, so I understand in principle.

So, there are a few other options here.

1. Add these locations as additional artifact directories, so that the docker-worker uploads them for you (no need to copy them first).  This is probably the cleanest.

2. Modify to perform this copy after the Mozharness run, regardless of its success, preferably in a more generic way.  Perhaps by setting POST_RUN_ARTIFACT_COPIES to a space-separated list of source, destination directories?

3. `build/` already supports uploading to an artifact directory, so you could just enable uploading and configure these directories appropriately:
Attachment #8722886 - Flags: review?(dustin)
Attachment #8722886 - Attachment is obsolete: true
Comment on attachment 8722885 [details]
MozReview Request: Bug 1249166 - Part 1: Add post-build mach commands to mozharness. r?jlund

if it works for you in testing/try, it looks good to me. :)

::: testing/mozharness/mozharness/mozilla/building/
(Diff revision 1)
> +        # A list of argument lists.  Better names gratefully accepted!

names seem fine to me! I really like this extendable approach..
Attachment #8722885 - Flags: review?(jlund) → review+
Comment on attachment 8723116 [details]
MozReview Request: Bug 1249166 - Part 2: Copy Gradle artifacts. r?dustin
Attachment #8723116 - Flags: review?(dustin) → review+
Assignee: nobody → nalexander
Product: Firefox for Android → Firefox Build System
Target Milestone: Firefox 47 → mozilla47
You need to log in before you can comment on or make changes to this bug.