1581979 - Use SNAPSHOT build versions for local GeckoView builds

Status: RESOLVED FIXED

(Firefox Build System :: Android Studio and Gradle Integration, defect, P3)

Description

[Nick Alexander :nalexander [he/him]]

The local GeckoView substitution added in Bug 1533465 appears not to work as well as I suggested in the comment. In FirefoxReality, at least, ./mach build produces a fresh AAR but Gradle doesn't consume it. What's happening is:

  63081:14:52:49.552 [DEBUG] [org.gradle.api.internal.artifacts.repositories.resolver.DefaultExternalResourceArtifactResolver] Loading file:/Users/nalexander/Mozilla/objdirs/objdir-droid/gradle/build/mobile/android/geckoview/maven/org/mozilla/geckoview/geckoview-default/71.0.20190917144005/geckoview-default-71.0.20190917144005.aar
...
  75548:14:52:50.008 [INFO] [org.gradle.api.internal.artifacts.transform.TransformationStep] Transforming artifact geckoview-default.aar (org.mozilla.geckoview:geckoview-default:71.0.20190917144005) with JetifyTransform
  75549:14:52:50.341 [INFO] [org.gradle.internal.execution.steps.ResolveCachingStateStep] Caching disabled for JetifyTransform: /Users/nalexander/Mozilla/objdirs/objdir-droid/gradle/build/mobile/android/geckoview/maven/org/mozilla/geckoview/geckoview-default/71.0.20190917144005/geckoview-default-71.0.20190917144005.aar because:
  75551:14:52:50.342 [DEBUG] [org.gradle.internal.execution.steps.SkipUpToDateStep] Determining if JetifyTransform: /Users/nalexander/Mozilla/objdirs/objdir-droid/gradle/build/mobile/android/geckoview/maven/org/mozilla/geckoview/geckoview-default/71.0.20190917144005/geckoview-default-71.0.20190917144005.aar is up-to-date
  75552:14:52:50.343 [INFO] [org.gradle.internal.execution.steps.SkipUpToDateStep] Skipping JetifyTransform: /Users/nalexander/Mozilla/objdirs/objdir-droid/gradle/build/mobile/android/geckoview/maven/org/mozilla/geckoview/geckoview-default/71.0.20190917144005/geckoview-default-71.0.20190917144005.aar as it is up-to-date.
...
  79225:14:52:50.553 [DEBUG] [org.gradle.internal.execution.steps.SkipUpToDateStep] Determining if ExtractAarTransform: /Users/nalexander/.gradle/caches/transforms-2/files-2.1/88598f3fd7bca59541a4f1bfbabf061b/jetified-geckoview-default-71.0.20190917144005.aar is up-to-date
  79226:14:52:50.553 [INFO] [org.gradle.internal.execution.steps.SkipUpToDateStep] Skipping ExtractAarTransform: /Users/nalexander/.gradle/caches/transforms-2/files-2.1/88598f3fd7bca59541a4f1bfbabf061b/jetified-geckoview-default-71.0.20190917144005.aar as it is up-to-date.

This could have changed with Gradle versions or could always have been incorrect, although I definitely tested this scenario at some point.

The general solution is to use -SNAPSHOT version names for GeckoView. That works just fine, and addresses this issue. However, it leaves -SNAPSHOT AARs in the Gradle build directory, and for me those are 50Mb a piece. Disk space quickly dies, and appears to never get reclaimed (without a clobber or similar). Worse, Gradle's internal .caches directories mirror the explosion in verisons, although those at least appear to get cleaned up (albeit very slowly).

Comment 1

[Nick Alexander :nalexander [he/him]]

rbarker: ultimately, this is a choice the GV team and downstream consumers must make: is the disk space and lack of transparency around it worth it? I don't see ways to make Gradle invalidate bits more aggressively, sadly.

You could test locally with the patch I just submitted to see if that addresses the issue for you.

Comment 2

[Nick Alexander :nalexander [he/him]]

Attachment: Bug 1581979 - Use -SNAPSHOT versions locally to enable the local GeckoView substitution flow.

Comment 3

[Randall Barker [:rbarker]]

Personally I have a large disk and unfortunately clobber my gecko build often enough that I don't think it will be too large of an issue for me. But other people with more constrained disk space might have a problem. The gradle cache might also be an issue since there doesn't seem to be a good way to manually manage it.

Comment 4

[BugBot [:suhaib / :marco/ :calixte]]

The priority flag is not set for this bug.
:nalexander, could you have a look please?

For more information, please visit auto_nag documentation.

Comment 5

[Nick Alexander :nalexander [he/him]]

(In reply to Release mgmt bot [:sylvestre / :calixte / :marco for bugbug] from comment #4)

The priority flag is not set for this bug.
:nalexander, could you have a look please?

I'm going to say this is P3 because I don't actually know if this is a problem in the wild. Some people seem to have issues; some people seem to use the existing system without problems. I conjecture that this is either Gradle version or Android-Gradle plugin version dependent.

Comment 6

[Hiroyuki Ikezoe (:hiro)]

I am probably running into an issue related to this bug.

I am going to add a new API in GeckoView in bug 1586144, with local m-c builds exposing the new API I get an Unresolve reference: <the new API> error when I build reference-browser with local android-component.

For references for persons who are struggling with the same issue; A workaround I just realized is to replace the latest nightly aar in $HOME/.gradle/caches/modules-2/files-2.1/org.mozilla.geckoview/geckoview-nightly/ with local build geckoview-nightly aar.

Comment 7

[Botond Ballo [:botond]]

I've run into this as well. It worked the first time I built reference-browser against a custom GeckoView, but on a later occasion, after I updated the reference-browser repo and built a new GV, the build was still using my old GV until manually deleting the cache.

Comment 8

[Timothy Nikkel (:tnikkel)]

People keep running into this. Can you maybe mention this somewhere in the build docs or somewhere so more people don't get burned by this?

Comment 10

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/11e43c60856b
Use -SNAPSHOT versions locally to enable the local GeckoView substitution flow. r=nalexander

Comment 11

[Cosmin Sabou [:CosminS]]

https://hg.mozilla.org/mozilla-central/rev/11e43c60856b

Comment 12

[Hiroyuki Ikezoe (:hiro)]

\o/