1625126 - mobile: Reduce network usage in build tasks

Status: RESOLVED FIXED

(Release Engineering :: General, enhancement)

Description

[Johan Lorenzo [:jlorenzo]]

Since the taskgraph conversion (bug 1580778), we've increased the number of network requests done whenever a gradle project builds. Before then, we had this hack in the docker image: we embedded the gradle cache[1] so that the number of dependencies to fetch would be reduce to only newer ones.

In bug 1622339, we're moving the android-sdk away from the docker image. Let's reuse that logic to create a gradle cache that build task can use.

Ideally, we would backport what we have in mozilla-central[2]. That is to say: we spin a toochain task that hosts a maven proxy (nexus)[3] and let it fetch all deps[4], then expose the proxy as local files[5]. Then, the builds tasks will only use these local files to build[6]. I think we want to go there eventually. But sadly the logic relies on gradle being driven by mach[7]. In mobile projects, we just have gradle and I'd need to investigate how to change gradle to use these nexus local files instead.

I know the current state of the build tasks is hard to manage and the mobile teams want to have quick results so they don't have to rerun these intermittent build tasks. Thus, I'm going to start with something simpler, that still leverages bug 1622339

[1] https://github.com/mozilla-mobile/fenix/blob/6710e4604dcb8e952db3230baa528225153b100a/automation/docker/Dockerfile#L82-L85
[2] https://firefox-source-docs.mozilla.org/build/buildsystem/toolchains.html#firefox-for-android-with-gradle
[3] https://searchfox.org/mozilla-central/rev/4d9cd186767978a99dafe77eb536a9525980e118/taskcluster/scripts/misc/android-gradle-dependencies/before.sh#14
[4] https://searchfox.org/mozilla-central/rev/4d9cd186767978a99dafe77eb536a9525980e118/taskcluster/scripts/misc/android-gradle-dependencies.sh#14
[5] https://searchfox.org/mozilla-central/rev/4d9cd186767978a99dafe77eb536a9525980e118/taskcluster/scripts/misc/android-gradle-dependencies/after.sh#28
[6] https://searchfox.org/mozilla-central/rev/4d9cd186767978a99dafe77eb536a9525980e118/mobile/android/config/mozconfigs/common#25
[7] https://searchfox.org/mozilla-central/rev/4d9cd186767978a99dafe77eb536a9525980e118/build.gradle#29 and https://searchfox.org/mozilla-central/rev/4d9cd186767978a99dafe77eb536a9525980e118/mobile/android/gradle.configure#337

Comment 1

[Johan Lorenzo [:jlorenzo]]

Attachment: Bug 1625126 - Enable toolchain tasks to fetch from other toolchain tasks r=tomprince

Comment 2

[Johan Lorenzo [:jlorenzo]]

Attachment: [fenix] Cache external dependencies in a single task and let gradle tasks use it

Comment 3

[Johan Lorenzo [:jlorenzo]]

Attachment: [android-components] Cache external dependencies in a single task and let gradle tasks use it

Comment 4

[Johan Lorenzo [:jlorenzo]]

Attachment: [reference-browser] Cache external dependencies in a single task and let gradle tasks use it

Comment 5

[Johan Lorenzo [:jlorenzo]]

Attachment: [fenix] Cache external dependencies in a single task and let gradle tasks use it

Comment 6

[Johan Lorenzo [:jlorenzo]]

Each piece has landed. Followup work will be done in bug 1637891.

Comment 7

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/17947364e4d7
[components] Bug 1625126 - Cache external dependencies in a single task and let gradle tasks use it
https://hg.mozilla.org/integration/autoland/rev/2048b13f9e7f
[fenix] Bug 1625126 - Cache external dependencies in a single task and let gradle tasks use it (https://github.com/mozilla-mobile/fenix/pull/10316)