1217228 - Investigate "fast-path" for installing "build faster" changes to Fennec using Gradle

Status: NEW

(Firefox Build System :: Android Studio and Gradle Integration, defect, P5)

Description

[Nick Alexander :nalexander [he/him]]

This is a Fennec-specific version of Facebook's exopackage: we can detect builds that only touch the omnijar and push just those changes to the device.

I had this working in Eclipse using dependent APKs, but I don't see a way to add such a "runtime APK" dependency using Gradle.

Here's a potential implementation strategy:
* find a way to /not/ rebuild the APK in the face of omnijar-only changes;
* push the updated omnijar to the device;
* update https://dxr.mozilla.org/mozilla-central/source/xpcom/build/Omnijar.cpp#50 and https://dxr.mozilla.org/mozilla-central/source/mobile/android/base/util/GeckoJarReader.java#265 to look in more locations;
** perhaps only for !MOZILLA_OFFICIAL builds;
** perhaps comparing timestamps to avoid stale artifacts;
* find a way to kill the browser safely, so the fresh omnijar is picked up at restart.

Comment 1

[Mike Hommey [:glandium]]

You shouldn't have to touch Omnijar.cpp. You should be able to pass a different -greomni pass when initializing gecko (see mobile/android/base/GeckoThread.java)

Comment 2

[Nick Alexander :nalexander [he/him]]

friedger: are you familiar with Facebook's exopackage?  That uses multi-dex and some build system cleverness (built into Buck) to "hot swap" the majority of compiled Java code in the App without pushing a whole new APK to device.  For Java-only changes, it's much faster than using |adb install| or equivalents.

Doing that is hard for Fennec for historical reasons.  However, we have this big chunk of "non-Java, non-Android" code (JavaScript code and resources) that is built into "the omnijar".  If we make changes to just that part, we could push just the new omnijar to the device and use it rather than the one in the APK itself.

This is a general problem, of course: lots of Apps have data files that one might want to rev independently of the APK.  It's tricky to push files in this way and keep track of what's really on the device, though.  Have you ever heard of systems that do this?  Do you have any thoughts on building a Gradle plugin that could do this?  (I have some notes on desired features in the first comment.)

Thanks!

Comment 3

[friedger]

nick: could you explain a bit more what was working before with eclipse? And what you would like to improve?

After an initial build (`gw installDebug`) the build with changes only in omnijar currently takes on my machine 1min 30s, or so. Is this what you would like to get down?

The process would be to push the omnijar via gradle to the sdcard or so of the devices and then the jarReader would look for the jar on the sdcard or so, correct? Is sdcard a possible location?

For stopping the browser one could use something like:
`adb shell am force-stop <packagename>`

One could remove the omnijar dependency from the app module and use just a runtime artifact dependency.

Comment 4

[Nick Alexander :nalexander [he/him]]

Hi Friedger!

(In reply to friedger from comment #3)
> nick: could you explain a bit more what was working before with eclipse?

I had a "runtime dependency" working, like you describe below.  That is, I had a Fennec APK that runtime-depended on an Omnijar APK, and the IDE managed pushing just the changed APKs to device.

 And
> what you would like to improve?

> After an initial build (`gw installDebug`) the build with changes only in
> omnijar currently takes on my machine 1min 30s, or so. Is this what you
> would like to get down?

Yes, exactly.  If we can do this with a "runtime dependency", it might look like building an Omnijar APK quickly, and then arranging to only re-install that APK on device.
 
> The process would be to push the omnijar via gradle to the sdcard or so of
> the devices and then the jarReader would look for the jar on the sdcard or
> so, correct? Is sdcard a possible location?

Correct.  And yes, SD card is a possible location.  (I used a separate APK before 'cuz the IDE handled pushing the right APKs across.  That's the bit that's tricky.) 

> For stopping the browser one could use something like:
> `adb shell am force-stop <packagename>`
> 
> One could remove the omnijar dependency from the app module and use just a
> runtime artifact dependency.

If you know how to do this with Gradle, that would be *awesome*.  The omnijar is not in any way Android specific -- a proof of concept Gradle project with a text file would be extremely helpful.  Thanks!

Comment 5

[friedger]

Hi Nick,

(In reply to Nick Alexander :nalexander from comment #4)

> If you know how to do this with Gradle, that would be *awesome*.  The
> omnijar is not in any way Android specific -- a proof of concept Gradle
> project with a text file would be extremely helpful.  Thanks!

I have created a small project at https://github.com/friedger/runtime-dependency that "prepares" a text file in one module and has an Android app reading the text file in the other app. There is no compile dependency between the modules.

Does it make sense to you?

Comment 6

[Nick Alexander :nalexander [he/him]]

(In reply to friedger from comment #5)
> Hi Nick,
> 
> (In reply to Nick Alexander :nalexander from comment #4)
> 
> > If you know how to do this with Gradle, that would be *awesome*.  The
> > omnijar is not in any way Android specific -- a proof of concept Gradle
> > project with a text file would be extremely helpful.  Thanks!
> 
> I have created a small project at
> https://github.com/friedger/runtime-dependency that "prepares" a text file
> in one module and has an Android app reading the text file in the other app.
> There is no compile dependency between the modules.
> 
> Does it make sense to you?

Thank you for investigating!  It does, or at least mostly.  I'm a little confused as to what the extra configuration really achieves.  It appears to be your hook into Gradle's task dependency tree.  Is it necessary?

I have a couple of questions:

1) did you investigate Gradle APK runtime dependencies?  Like "provided scope", in Maven?

2) My real goal is to have this work automatically from IntelliJ.  I don't think that IJ uses the Gradle install* targets to deploy APKs; I think they do their own thing.  Any thoughts on how to make this "just work" with IntelliJ?  (Or Android Studio.)

In any case, this is awesome, and I learned a few things.  (You definitely know more Gradle than I do!)

And I have a couple of next steps:

1) If you can, would you look at the Gradle configuration rooted in the Mozilla topsrcdir and think a little about how this might work with the Fennec omnijar?  The omnijar project in mobile/android/app/omnijar is a lie; it exists to label stuff in IntelliJ.  (See comments.)  The real work is in https://dxr.mozilla.org/mozilla-central/source/mobile/android/app/build.gradle#98https://dxr.mozilla.org/mozilla-central/source/mobile/android/app/build.gradle#98, and the Makefile.in target is at https://dxr.mozilla.org/mozilla-central/source/mobile/android/base/Makefile.in?from=android%2Fbase%2FMakefile.in#477.

2) Can you think a little bit about how the compiled App might make sure it doesn't use a stale omnijar in /mnt/sdcard?  I'm thinking of maybe pushing to /mnt/sdcard/fennec_nalexander/HASH/assets/omni.ja or something like that, and trying to make sure the HASH is always correct.  This can be tricky.

Comment 7

[friedger]

(In reply to Nick Alexander :nalexander from comment #6)

> Thank you for investigating!  It does, or at least mostly.  I'm a little
> confused as to what the extra configuration really achieves.  It appears to
> be your hook into Gradle's task dependency tree.  Is it necessary?

It looks like it is not necessary (I am not a Gradle expert either):
https://github.com/friedger/runtime-dependency/commit/e1c1233f34d84f0da1bc25d2c2d7b9696fc9124d
Now, we only have a dependency in the tasks. Do we need more?

> 
> I have a couple of questions:
> 
> 1) did you investigate Gradle APK runtime dependencies?  Like "provided
> scope", in Maven?
What I understand e.g. from a a plugin that tries to mimic the provided scope (https://github.com/spring-projects/gradle-plugins/tree/master/propdeps-plugin) is that they basically create a new configuration.

> 
> 2) My real goal is to have this work automatically from IntelliJ.  I don't
> think that IJ uses the Gradle install* targets to deploy APKs; I think they
> do their own thing.  Any thoughts on how to make this "just work" with
> IntelliJ?  (Or Android Studio.)
They use assemble and then run a shell script. So, the following should work:
https://github.com/friedger/runtime-dependency/commit/9cd7db1a9da086f3ebdff65b222aa57e3b79c12f


> 1) If you can, would you look at the Gradle configuration rooted in the
> Mozilla topsrcdir and think a little about how this might work with the
> Fennec omnijar?  
I'll look at that.

> 2) Can you think a little bit about how the compiled App might make sure it
> doesn't use a stale omnijar in /mnt/sdcard? 
Stale means that a new omnijar was pushed to the device but fennec not stopped? Or what is the scenario where things could go wrong?

Comment 8

[friedger]

(In reply to Nick Alexander :nalexander from comment #6)
> 1) If you can, would you look at the Gradle configuration rooted in the
> Mozilla topsrcdir and think a little about how this might work with the
> Fennec omnijar? 

Not sure whether I understood what you mean by "this" mean. Something like

task pushOmnijar(type: Files) {
    script {
        File file = project(":omnijar").tasks.getByName('buildOmnijar').outputs.files.singleFile
        push file.getAbsolutePath(), '/sdcard/' + file.getName()
        println "pushed omnijar " + this.name
    }
}

Comment 9

[Nick Alexander :nalexander [he/him]]

(In reply to friedger from comment #8)
> (In reply to Nick Alexander :nalexander from comment #6)
> > 1) If you can, would you look at the Gradle configuration rooted in the
> > Mozilla topsrcdir and think a little about how this might work with the
> > Fennec omnijar? 
> 
> Not sure whether I understood what you mean by "this" mean. Something like
> 
> task pushOmnijar(type: Files) {
>     script {
>         File file =
> project(":omnijar").tasks.getByName('buildOmnijar').outputs.files.singleFile
>         push file.getAbsolutePath(), '/sdcard/' + file.getName()
>         println "pushed omnijar " + this.name
>     }
> }

friedger: sorry to not get back to you for so long; Mozilla convened in Orlando at the beginning of December and I'm only just now getting back into the regular routine -- just in time for Christmas vacation.

I see what you're doing with the explicit push to the device; I'll try to think a little about making this automatic from within the IDE.  We may just find this is solved as Google builds out the Instant Run feature, and eventually handles updating Android assets automatically.  Thanks again for your work here!

Comment 10

[Firefox Bug Husbandry Bot]

Re-triaging per https://bugzilla.mozilla.org/show_bug.cgi?id=1473195

Needinfo :susheel if you think this bug should be re-triaged.