1247170 - Add Android Gradle dependency fetching docker container

Status: RESOLVED FIXED

(Firefox Build System :: Android Studio and Gradle Integration, defect)

Description

[Nick Alexander :nalexander [he/him] (PTO Feb 24-March 3)]

This is just like Bug 1237366, but for fetching Android Gradle dependencies.

The goal is to have a Docker container defined in tree, and a consuming Taskcluster task, which together fetch Android Gradle dependencies.  I see three parts to this:

1) Gradle itself;
2) the Android SDK;
3) Gradle dependencies, like the toolchain plugins, but also consumed libraries for testing, etc.

For 1) There's no automation consumer of Gradle (yet -- see Bug 1119520 and friends).  For 2) upgrading the Android toolchain is a manual process conducted on a local machine and uploaded to tooltool.  I'd like to make this automatic, but it's not the focus of this work.  There are redistribution and tooltool integration issues in the way (see Bug 1245170).

3) is what I really care about.  I have a challenging manual process for this which involves running a proxy Nexus Maven Repository and copying things around.  The goal here is to improve this.

Comment 1

[Nick Alexander :nalexander [he/him] (PTO Feb 24-March 3)]

Attachment: MozReview Request: Bug 1247170 - Part 1: Add "android-gradle-build" Docker image. r=dustin,sebastian r?gps

We can't run Docker-in-Docker, so this is a clone of
https://hub.docker.com/r/sonatype/nexus, modified to run additional
commands.  Sonatype Nexus officially supports the Oracle JDK, so we
use that; eventually we can test against the OpenJDK and try to use
that instead.

This bakes a build.sh and Gradle-specific dependencies into the docker
image.  This makes sense in a future where the dependencies this
fetches are baked into the image used to build Fennec.  It makes less
sense right now, when the dependencies are uploaded to tooltool and
subsequently consumed.  It's not critical that we get this right
immediately.

The actual build.sh script runs the Gradle wrapper for a special
dependency project.  The wrapper installs Gradle itself, downloads a
large number of Gradle dependencies, and uses Jake Wharton's
https://github.com/JakeWharton/sdk-manager-plugin/ to download and
install the Android SDK.  (We could use |mach bootstrap|, but that
doesn't (yet) support non-interactive use, and I want to try to use
the SDK Manager more generally.)  These outputs are then packaged for
further use: right now, they're exposed as artifacts, but eventually
we'll upload them directly to tooltool.

Review commit: https://reviewboard.mozilla.org/r/34301/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/34301/

Comment 2

[Nick Alexander :nalexander [he/him] (PTO Feb 24-March 3)]

Attachment: MozReview Request: Bug 1247170 - Part 2: Add "android-gradle-build-dependencies" Taskcluster job. r=dustin

Review commit: https://reviewboard.mozilla.org/r/34303/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/34303/

Comment 3

[Nick Alexander :nalexander [he/him] (PTO Feb 24-March 3)]

dustin: you saw a related (but different) part of my approach in https://reviewboard.mozilla.org/r/30445/#issue-summary.

This is specifically about fetching dependencies; build tasks that use the produced artifacts to come.

sebastian: you're here to sanity check the Gradle definition, to comment on the approach, and to comment on SDK Manager.  (Links in code.)

Comment 4

[Nick Alexander :nalexander [he/him] (PTO Feb 24-March 3)]

https://treeherder.mozilla.org/#/jobs?repo=try&revision=467ebfdc1bde

Comment 5

[Dustin J. Mitchell [:dustin] (he/him)]

Comment on attachment 8717778 [details]
MozReview Request: Bug 1247170 - Part 2: Add "android-gradle-build-dependencies" Taskcluster job. r=dustin

https://reviewboard.mozilla.org/r/34303/#review31055

::: testing/taskcluster/tasks/tests/android-gradle-build-dependencies.yml:3
(Diff revision 1)
> +    from: 'tasks/lint.yml'

This seems pretty weird to me.  Especially since this file is in the tasks/tests directory.  Maybe a new task type and directory?

Comment 6

[Dustin J. Mitchell [:dustin] (he/him)]

Comment on attachment 8717777 [details]
MozReview Request: Bug 1247170 - Part 1: Add "android-gradle-build" Docker image. r=dustin,sebastian r?gps

https://reviewboard.mozilla.org/r/34301/#review31053

::: testing/docker/android-gradle-build/build.sh:14
(Diff revision 1)
> +  http://localhost:8081/nexus/service/local/status || true

May want to 'rm -f status' afterward

::: testing/docker/android-gradle-build/build.sh:44
(Diff revision 1)
> +# upload to (internal) tooltool eventually.  mv

I think we've talked about it, but just to be sure: it's possible to make TC artifacts that are non-public, too.

::: testing/docker/android-gradle-build/project/README.md:1
(Diff revision 1)
> +This Gradle project exists only to list dependencies used to build Firefox f or

typo "f or"

Comment 7

[Dustin J. Mitchell [:dustin] (he/him)]

https://reviewboard.mozilla.org/r/34303/#review31069

::: testing/taskcluster/tasks/tests/android-gradle-build-dependencies.yml:3
(Diff revision 1)
> +    from: 'tasks/lint.yml'

To elaborate a little bit: this isn't a lint job.  I think it would make the most sense to put this task dsecription directly in testing/taskcluster/tasks and not inherit from any other .yml.  Just manually incorporate the important bits from lint.yml.

Comment 8

[Nick Alexander :nalexander [he/him] (PTO Feb 24-March 3)]

https://treeherder.mozilla.org/#/jobs?repo=try&revision=1514df944822

Comment 9

[Nick Alexander :nalexander [he/him] (PTO Feb 24-March 3)]

https://treeherder.mozilla.org/#/jobs?repo=try&revision=bde3e1ed618d

Comment 10

[Nick Alexander :nalexander [he/him] (PTO Feb 24-March 3)]

https://treeherder.mozilla.org/#/jobs?repo=try&revision=f653056c6e98

Comment 11

[Nick Alexander :nalexander [he/him] (PTO Feb 24-March 3)]

Comment on attachment 8717777 [details]
MozReview Request: Bug 1247170 - Part 1: Add "android-gradle-build" Docker image. r=dustin,sebastian r?gps

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/34301/diff/1-2/

Comment 12

[Nick Alexander :nalexander [he/him] (PTO Feb 24-March 3)]

Comment on attachment 8717778 [details]
MozReview Request: Bug 1247170 - Part 2: Add "android-gradle-build-dependencies" Taskcluster job. r=dustin

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/34303/diff/1-2/

Comment 13

[Gregory Szorc [:gps]]

https://reviewboard.mozilla.org/r/34301/#review31147

::: testing/docker/android-gradle-build/build.sh:37
(Diff revision 2)
> +curl --fail --silent --location --retry 5 \
> +  --output gradle-2.7-all.zip \
> +  https://services.gradle.org/distributions/gradle-2.7-all.zip

This should probably use tooltool since 3rd party servers aren't reliable nor can they be trusted (tooltool lets us pin hashes to detect content tampering).

::: testing/docker/android-gradle-build/project/build.gradle:19
(Diff revision 2)
> +    classpath 'com.github.JakeWharton:sdk-manager-plugin:220bf7a88a7072df3ed16dc8466fb144f2817070'

Is this another 3rd party dependency? Given the rate at which GitHub has been offline lately, this makes me nervous.

::: testing/docker/android-gradle-build/project/gradlew:140
(Diff revision 2)
> +        (0) set -- ;;
> +        (1) set -- "$args0" ;;
> +        (2) set -- "$args0" "$args1" ;;
> +        (3) set -- "$args0" "$args1" "$args2" ;;
> +        (4) set -- "$args0" "$args1" "$args2" "$args3" ;;
> +        (5) set -- "$args0" "$args1" "$args2" "$args3" "$args4" ;;
> +        (6) set -- "$args0" "$args1" "$args2" "$args3" "$args4" "$args5" ;;
> +        (7) set -- "$args0" "$args1" "$args2" "$args3" "$args4" "$args5" "$args6" ;;
> +        (8) set -- "$args0" "$args1" "$args2" "$args3" "$args4" "$args5" "$args6" "$args7" ;;
> +        (9) set -- "$args0" "$args1" "$args2" "$args3" "$args4" "$args5" "$args6" "$args7" "$args8" ;;

lol

::: testing/docker/android-gradle-build/Dockerfile:29
(Diff revision 2)
> +  http://download.oracle.com/otn-pub/java/jdk/7u76-b13/jdk-7u76-linux-x64.rpm \

Insecure download?!

Comment 14

[Nick Alexander :nalexander [he/him] (PTO Feb 24-March 3)]

https://reviewboard.mozilla.org/r/34301/#review31147

> This should probably use tooltool since 3rd party servers aren't reliable nor can they be trusted (tooltool lets us pin hashes to detect content tampering).

The whole point is to fetch and get this into tooltool.  I had intended to have the build use tooltool to download this and then unpack it so that Gradle is available to the build environment, but I may be able to have the build just use the Gradle wrapper directly (downloaded from the tooltool archive).  The Gradle wrapper can include a SHA256 hash, which will the desired hash guarantee.  (And will assert hashes for local builders too!)

> Is this another 3rd party dependency? Given the rate at which GitHub has been offline lately, this makes me nervous.

The whole point of this is to fetch deps from upstream.  It's a manually instigated job which will produce outputs for use via tooltool.  If GitHub or jcentral or some other upstream repository is down, that's fine.

> lol

For the record, this is not my code.  I assume this is to be compatible across shells and systems...

> Insecure download?!

Straight from https://hub.docker.com/r/sonatype/nexus/.

Comment 15

[Dustin J. Mitchell [:dustin] (he/him)]

We have several of these "load stuff into tooltool/artifacts" jobs now, which I think provide a good flexible buffer against external downtime.  If gradle.org is down, we just can't repackage gradle -- we can keep using the version in tooltool.  So this is a good model.

Why am I not surprised Oracle is the one who hasn't figured out how to SSL yet? :)

Comment 16

[Sebastian Kaspari (:sebastian; :pocmo)]

Comment on attachment 8717777 [details]
MozReview Request: Bug 1247170 - Part 1: Add "android-gradle-build" Docker image. r=dustin,sebastian r?gps

https://reviewboard.mozilla.org/r/34301/#review31219

Comment 17

[Dustin J. Mitchell [:dustin] (he/him)]

Comment on attachment 8717778 [details]
MozReview Request: Bug 1247170 - Part 2: Add "android-gradle-build-dependencies" Taskcluster job. r=dustin

https://reviewboard.mozilla.org/r/34303/#review31221

::: testing/taskcluster/tasks/android-gradle-build-dependencies.yml:28
(Diff revision 2)
> +    # the board.

omit this comment

Comment 18

[Nick Alexander :nalexander [he/him] (PTO Feb 24-March 3)]

Comment on attachment 8717777 [details]
MozReview Request: Bug 1247170 - Part 1: Add "android-gradle-build" Docker image. r=dustin,sebastian r?gps

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/34301/diff/2-3/

Comment 19

[Nick Alexander :nalexander [he/him] (PTO Feb 24-March 3)]

Comment on attachment 8717778 [details]
MozReview Request: Bug 1247170 - Part 2: Add "android-gradle-build-dependencies" Taskcluster job. r=dustin

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/34303/diff/2-3/

Comment 20

[Pulsebot]

https://hg.mozilla.org/integration/fx-team/rev/123c802d928a
https://hg.mozilla.org/integration/fx-team/rev/2b079941a33c

Comment 21

[Nick Alexander :nalexander [he/him] (PTO Feb 24-March 3)]

Comment on attachment 8717777 [details]
MozReview Request: Bug 1247170 - Part 1: Add "android-gradle-build" Docker image. r=dustin,sebastian r?gps

Removing gps's review; on IRC, he deferred to dustin.  I believe I addressed gps's concerns, which had to do with downloading from insecure sites, in the final ticket.

Comment 22

[Pulsebot]

Backout:
https://hg.mozilla.org/integration/fx-team/rev/0696fef38f0f

Comment 23

[Pulsebot]

https://hg.mozilla.org/integration/fx-team/rev/f4c2c933d74a
https://hg.mozilla.org/integration/fx-team/rev/3cc50d3b9063

Comment 24

[Carsten Book [:Tomcat]]

https://hg.mozilla.org/mozilla-central/rev/f4c2c933d74a
https://hg.mozilla.org/mozilla-central/rev/3cc50d3b9063