1509539 - Add gradle support for building gecko binaries

Status: NEW

(Firefox Build System :: Android Studio and Gradle Integration, enhancement, P3)

Description

[Nick Alexander :nalexander [he/him]]

+++ This bug was initially created as a clone of Bug #1418464 +++

I accidentally stole Bug 1418464 for just a piece of this work, so I'm cloning and updating that description:

It's nice that we have an Android Studio project that allow us to build geckoview_example and geckoview, but it would be nice if hitting the 'run' button would also bring in any changes to gecko itself. The workflow is basically:

1) Add a new method to GeckoView
2) Implement above method in Gecko using JS or native code
3) Write a test or new API in GeckoViewExample or other sibling project
4) Win

Since setting up a gecko build is such an involved process I think we could just punt on that. We'd probably just assume that 'mach build && mach package' works. I don't think we really want Studio to install a mozconfig, for instance. The instructions for getting this to work for developers would basically be 'mach bootstrap', set up mozconfig, then ensure 'mach configure' works.

Comment 1

[Nick Alexander :nalexander [he/him]]

Attachment: Bug 1509539 - Part 1a: Invoke |mach build binaries| during Gradle builds.

This is tricky because there's a dependency chain:

mach stage-package (building the omnijar)
  depends on
mach build binaries (building libxul.so)
  depends on
generating JNI wrappers
  depends on
compiling Java

Now, in Bug 1444546 we moved the JNI wrapper generation into the
Gradle build, and we made a JNI wrapper generating task for each
variant.  That's a problem because there really is only one JNI
wrapper generation option -- whatever is invoked as part of |mach
build| -- and because the proliferation of tasks makes |mach
stage-package| depend on lots of _different_ compiling Java tasks
(duplicating tons of work and slowing builds down significantly).

This ticket narrows us down to two JNI wrapper generating tasks.  The
task names are complicated in order to not change too many things.

Depends on D12799

Comment 2

[Nick Alexander :nalexander [he/him]]

Attachment: Bug 1509539 - Part 1b: Drop geckoBinaries flavor dimension. r?snorp

This is just the mechanics of dropping the flavor -- no real
consideration for whether it works yet.  These parts will all be
folded before landing.

Depends on D12800

Comment 3

[Nick Alexander :nalexander [he/him]]

Attachment: Bug 1509539 - Part 1c: Update automation. r?snorp

Again, these will all be folded together.

Depends on D12801

Comment 4

[Nick Alexander :nalexander [he/him]]

Attachment: Bug 1509539 - Part 1d: Make Gecko binaries depend on the Gradle task graph. r?snorp

This commit is the meat of the series.  (Remember, the series will all
land together; the parts are just for ease of reviewing.)

This moves the "with/without" Gecko binaries logic to depend on the
actual tasks invoked by Gradle.  If we are asked to `assemble`
GeckoView or Fennec (and we're not in the special circumstance of
|mach build|), then we are in the "withGeckoBinaries" flow:
- We invoke |mach build binaries| if COMPILE_ENVIRONMENT
- We invoke |make -C ... stage-package|
- We copy omni.ja and libs/ into place for the Android-Gradle AAR/APK packager

If we're not asked to `assemble` (or if we're under |mach build|, then
we don't do any of that.  We leave any omni.ja and libs/ in place, but
we don't require them.

If we get everything right, this somewhat transparently work the
moz.build system into the Gradle build system when it's required.  I'm
sure there will be issues, but we can start from this and work out
problems as they become clear.

Depends on D12802

Comment 5

[BugBot [:suhaib / :marco/ :calixte]]

There are some r+ patches which didn't land and no activity in this bug for 2 weeks.
:nalexander, could you have a look please?

Comment 6

[Nick Alexander :nalexander [he/him]]

(In reply to Release mgmt bot [:sylvestre / :calixte / :marco for bugbug] from comment #5)

There are some r+ patches which didn't land and no activity in this bug for 2 weeks.
:nalexander, could you have a look please?

Just circling back: this will take a different direction at some point, post Bug 1550146. snorp is actively looking at removing withGeckoBinaries, following these patches.

Comment 7

[Nick Alexander :nalexander [he/him]]

Note to self: moons ago, I worked out an approach for getting rid of withGeckoBinaries. That's in the patch series here. It's sneaky: it uses the Gradle task graph to determine if the Gecko binaries (libxul.so and friends, omni.ja) are needed.

But the moon has moved, and I think this sneakiness might not be necessary. The signal that Gecko binaries are needed should just be COMPILE_ENVIRONMENT || MOZ_ARTIFACT_BUILDS. If I'm correct, we could dramatically simplify this situation by using that as the conditional rather than having the geckoBinaries product flavour dimension.

Comment 8

[Fatih Kilic [:fkilic]]

I also need this feature. Running mach build then mach build faster adds about 2 extra minutes per compilation (I'm assuming it is because I'm running TIER: android-stage-package android-archive-geckoview twice, one in fast and one in full/binary build). Just running mach build works for me though, so I just removed args 'faster' from https://searchfox.org/mozilla-central/rev/535119812a0eda6f252f6378fbdbe46efa95970a/build.gradle#350 and it seems to be working for me.

Comment 9

[Nick Alexander :nalexander [he/him]]

(In reply to Fatih Kilic from comment #8)

I also need this feature. Running mach build then mach build faster adds about 2 extra minutes per compilation (I'm assuming it is because I'm running TIER: android-stage-package android-archive-geckoview twice, one in fast and one in full/binary build). Just running mach build works for me though, so I just removed args 'faster' from https://searchfox.org/mozilla-central/rev/535119812a0eda6f252f6378fbdbe46efa95970a/build.gradle#350 and it seems to be working for me.

Thanks! You identified the piece that I had in mind that we wouldn't want to run twice -- the android-* tiers. I can see at least two ways to make progress around this:

1. When invoking mach build from Gradle set MACH_INVOKED_WITHIN_GRADLE=1 and filter out the tiers we don't want when that is set around https://searchfox.org/mozilla-central/rev/535119812a0eda6f252f6378fbdbe46efa95970a/config/baseconfig.mk#57-60.
2. Invoke mach build binaries after mach build faster from within Gradle.

The main reason to maybe do 2. is that it's not clear to me what doing 1. will do when there are mach configure changes pending. A smaller reason is that a no-op mach build binaries is really quite fast, where-as a no-op mach build is really quite slow. It's not clear to me that binaries; faster is enough, but I think it is.

For the future: I wonder how close we can get to a "Gradle-powered cold start". That is, can we place a mozconfig and then open m-c in Android Studio and have that operation drive the mach configure, etc needed to completely build the tree? I don't think that's possible with only mach build binaries.

Comment 10

[Nick Alexander :nalexander [he/him]]

As a first step, a patch that also invokes mach build binaries after mach build faster around https://searchfox.org/mozilla-central/rev/4a8bd8efdfaa43dd14a16d3cb15bf86796fd1def/build.gradle#365 (only when we have a compile environment!) would be interesting. This isn't quite the same as mach build, but it should be quite a bit faster for no-op builds.