Closed Bug 1247170 Opened 4 years ago Closed 4 years ago

Add Android Gradle dependency fetching docker container

Categories

(Firefox Build System :: Android Studio and Gradle Integration, defect)

defect
Not set

Tracking

(firefox47 fixed)

RESOLVED FIXED
Tracking Status
firefox47 --- fixed

People

(Reporter: nalexander, Assigned: nalexander)

References

Details

Attachments

(2 files)

This is just like Bug 1237366, but for fetching Android Gradle dependencies.

The goal is to have a Docker container defined in tree, and a consuming Taskcluster task, which together fetch Android Gradle dependencies.  I see three parts to this:

1) Gradle itself;
2) the Android SDK;
3) Gradle dependencies, like the toolchain plugins, but also consumed libraries for testing, etc.

For 1) There's no automation consumer of Gradle (yet -- see Bug 1119520 and friends).  For 2) upgrading the Android toolchain is a manual process conducted on a local machine and uploaded to tooltool.  I'd like to make this automatic, but it's not the focus of this work.  There are redistribution and tooltool integration issues in the way (see Bug 1245170).

3) is what I really care about.  I have a challenging manual process for this which involves running a proxy Nexus Maven Repository and copying things around.  The goal here is to improve this.
We can't run Docker-in-Docker, so this is a clone of
https://hub.docker.com/r/sonatype/nexus, modified to run additional
commands.  Sonatype Nexus officially supports the Oracle JDK, so we
use that; eventually we can test against the OpenJDK and try to use
that instead.

This bakes a build.sh and Gradle-specific dependencies into the docker
image.  This makes sense in a future where the dependencies this
fetches are baked into the image used to build Fennec.  It makes less
sense right now, when the dependencies are uploaded to tooltool and
subsequently consumed.  It's not critical that we get this right
immediately.

The actual build.sh script runs the Gradle wrapper for a special
dependency project.  The wrapper installs Gradle itself, downloads a
large number of Gradle dependencies, and uses Jake Wharton's
https://github.com/JakeWharton/sdk-manager-plugin/ to download and
install the Android SDK.  (We could use |mach bootstrap|, but that
doesn't (yet) support non-interactive use, and I want to try to use
the SDK Manager more generally.)  These outputs are then packaged for
further use: right now, they're exposed as artifacts, but eventually
we'll upload them directly to tooltool.

Review commit: https://reviewboard.mozilla.org/r/34301/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/34301/
Attachment #8717777 - Flags: review?(s.kaspari)
Attachment #8717777 - Flags: review?(dustin)
dustin: you saw a related (but different) part of my approach in https://reviewboard.mozilla.org/r/30445/#issue-summary.

This is specifically about fetching dependencies; build tasks that use the produced artifacts to come.

sebastian: you're here to sanity check the Gradle definition, to comment on the approach, and to comment on SDK Manager.  (Links in code.)
Assignee: nobody → nalexander
Status: NEW → ASSIGNED
Comment on attachment 8717778 [details]
MozReview Request: Bug 1247170 - Part 2: Add "android-gradle-build-dependencies" Taskcluster job. r=dustin

https://reviewboard.mozilla.org/r/34303/#review31055

::: testing/taskcluster/tasks/tests/android-gradle-build-dependencies.yml:3
(Diff revision 1)
> +    from: 'tasks/lint.yml'

This seems pretty weird to me.  Especially since this file is in the tasks/tests directory.  Maybe a new task type and directory?
Attachment #8717778 - Flags: review?(dustin)
Comment on attachment 8717777 [details]
MozReview Request: Bug 1247170 - Part 1: Add "android-gradle-build" Docker image. r=dustin,sebastian r?gps

https://reviewboard.mozilla.org/r/34301/#review31053

::: testing/docker/android-gradle-build/build.sh:14
(Diff revision 1)
> +  http://localhost:8081/nexus/service/local/status || true

May want to 'rm -f status' afterward

::: testing/docker/android-gradle-build/build.sh:44
(Diff revision 1)
> +# upload to (internal) tooltool eventually.  mv

I think we've talked about it, but just to be sure: it's possible to make TC artifacts that are non-public, too.

::: testing/docker/android-gradle-build/project/README.md:1
(Diff revision 1)
> +This Gradle project exists only to list dependencies used to build Firefox f or

typo "f or"
Attachment #8717777 - Flags: review?(dustin) → review+
https://reviewboard.mozilla.org/r/34303/#review31069

::: testing/taskcluster/tasks/tests/android-gradle-build-dependencies.yml:3
(Diff revision 1)
> +    from: 'tasks/lint.yml'

To elaborate a little bit: this isn't a lint job.  I think it would make the most sense to put this task dsecription directly in testing/taskcluster/tasks and not inherit from any other .yml.  Just manually incorporate the important bits from lint.yml.
Comment on attachment 8717777 [details]
MozReview Request: Bug 1247170 - Part 1: Add "android-gradle-build" Docker image. r=dustin,sebastian r?gps

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/34301/diff/1-2/
Comment on attachment 8717778 [details]
MozReview Request: Bug 1247170 - Part 2: Add "android-gradle-build-dependencies" Taskcluster job. r=dustin

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/34303/diff/1-2/
Attachment #8717778 - Flags: review?(dustin)
https://reviewboard.mozilla.org/r/34301/#review31147

::: testing/docker/android-gradle-build/build.sh:37
(Diff revision 2)
> +curl --fail --silent --location --retry 5 \
> +  --output gradle-2.7-all.zip \
> +  https://services.gradle.org/distributions/gradle-2.7-all.zip

This should probably use tooltool since 3rd party servers aren't reliable nor can they be trusted (tooltool lets us pin hashes to detect content tampering).

::: testing/docker/android-gradle-build/project/build.gradle:19
(Diff revision 2)
> +    classpath 'com.github.JakeWharton:sdk-manager-plugin:220bf7a88a7072df3ed16dc8466fb144f2817070'

Is this another 3rd party dependency? Given the rate at which GitHub has been offline lately, this makes me nervous.

::: testing/docker/android-gradle-build/project/gradlew:140
(Diff revision 2)
> +        (0) set -- ;;
> +        (1) set -- "$args0" ;;
> +        (2) set -- "$args0" "$args1" ;;
> +        (3) set -- "$args0" "$args1" "$args2" ;;
> +        (4) set -- "$args0" "$args1" "$args2" "$args3" ;;
> +        (5) set -- "$args0" "$args1" "$args2" "$args3" "$args4" ;;
> +        (6) set -- "$args0" "$args1" "$args2" "$args3" "$args4" "$args5" ;;
> +        (7) set -- "$args0" "$args1" "$args2" "$args3" "$args4" "$args5" "$args6" ;;
> +        (8) set -- "$args0" "$args1" "$args2" "$args3" "$args4" "$args5" "$args6" "$args7" ;;
> +        (9) set -- "$args0" "$args1" "$args2" "$args3" "$args4" "$args5" "$args6" "$args7" "$args8" ;;

lol

::: testing/docker/android-gradle-build/Dockerfile:29
(Diff revision 2)
> +  http://download.oracle.com/otn-pub/java/jdk/7u76-b13/jdk-7u76-linux-x64.rpm \

Insecure download?!
https://reviewboard.mozilla.org/r/34301/#review31147

> This should probably use tooltool since 3rd party servers aren't reliable nor can they be trusted (tooltool lets us pin hashes to detect content tampering).

The whole point is to fetch and get this into tooltool.  I had intended to have the build use tooltool to download this and then unpack it so that Gradle is available to the build environment, but I may be able to have the build just use the Gradle wrapper directly (downloaded from the tooltool archive).  The Gradle wrapper can include a SHA256 hash, which will the desired hash guarantee.  (And will assert hashes for local builders too!)

> Is this another 3rd party dependency? Given the rate at which GitHub has been offline lately, this makes me nervous.

The whole point of this is to fetch deps from upstream.  It's a manually instigated job which will produce outputs for use via tooltool.  If GitHub or jcentral or some other upstream repository is down, that's fine.

> lol

For the record, this is not my code.  I assume this is to be compatible across shells and systems...

> Insecure download?!

Straight from https://hub.docker.com/r/sonatype/nexus/.
We have several of these "load stuff into tooltool/artifacts" jobs now, which I think provide a good flexible buffer against external downtime.  If gradle.org is down, we just can't repackage gradle -- we can keep using the version in tooltool.  So this is a good model.

Why am I not surprised Oracle is the one who hasn't figured out how to SSL yet? :)
Attachment #8717777 - Flags: review?(s.kaspari) → review+
Comment on attachment 8717777 [details]
MozReview Request: Bug 1247170 - Part 1: Add "android-gradle-build" Docker image. r=dustin,sebastian r?gps

https://reviewboard.mozilla.org/r/34301/#review31219
Comment on attachment 8717778 [details]
MozReview Request: Bug 1247170 - Part 2: Add "android-gradle-build-dependencies" Taskcluster job. r=dustin

https://reviewboard.mozilla.org/r/34303/#review31221

::: testing/taskcluster/tasks/android-gradle-build-dependencies.yml:28
(Diff revision 2)
> +    # the board.

omit this comment
Attachment #8717778 - Flags: review?(dustin) → review+
Comment on attachment 8717777 [details]
MozReview Request: Bug 1247170 - Part 1: Add "android-gradle-build" Docker image. r=dustin,sebastian r?gps

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/34301/diff/2-3/
Attachment #8717777 - Attachment description: MozReview Request: Bug 1247170 - Part 1: Add "android-gradle-build" Docker image. r?dustin,sebastian → MozReview Request: Bug 1247170 - Part 1: Add "android-gradle-build" Docker image. r=dustin,sebastian r?gps
Attachment #8717777 - Flags: review?(gps)
Attachment #8717778 - Attachment description: MozReview Request: Bug 1247170 - Part 2: Add "android-gradle-build-dependencies" Taskcluster job. r?dustin → MozReview Request: Bug 1247170 - Part 2: Add "android-gradle-build-dependencies" Taskcluster job. r=dustin
Comment on attachment 8717778 [details]
MozReview Request: Bug 1247170 - Part 2: Add "android-gradle-build-dependencies" Taskcluster job. r=dustin

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/34303/diff/2-3/
Comment on attachment 8717777 [details]
MozReview Request: Bug 1247170 - Part 1: Add "android-gradle-build" Docker image. r=dustin,sebastian r?gps

Removing gps's review; on IRC, he deferred to dustin.  I believe I addressed gps's concerns, which had to do with downloading from insecure sites, in the final ticket.
Attachment #8717777 - Flags: review?(gps)
https://hg.mozilla.org/mozilla-central/rev/f4c2c933d74a
https://hg.mozilla.org/mozilla-central/rev/3cc50d3b9063
Status: ASSIGNED → RESOLVED
Closed: 4 years ago
Resolution: --- → FIXED
Target Milestone: --- → mozilla47
Component: Build Config → Build Config & IDE Support
Product: Core → Firefox for Android
Target Milestone: mozilla47 → ---
Product: Firefox for Android → Firefox Build System
You need to log in before you can comment on or make changes to this bug.