1412006 - Migrate Android NDK to toolchain dependencies

Status: RESOLVED FIXED

(Firefox Build System :: Task Configuration, task)

Description

[Nick Alexander :nalexander [he/him]]

This ticket tracks turning the Android NDK into a proper toolchain task.  Right now, the Android NDK is manually (re)packed and uploaded to tooltool.  It's consumed like http://searchfox.org/mozilla-central/source/mobile/android/config/tooltool-manifests/android/releng.manifest#2-9.

Comment 1

[Nick Alexander :nalexander [he/him]]

froydnj: it's possible this will be obsoleted by building with clang on Android.  Is that correct?

Comment 2

[Nathan Froyd [:froydnj]]

My understanding is that building with clang on Android is orthogonal to whether the Android builds get their NDK via tooltool or the taskgraph.

Comment 3

[Nick Alexander :nalexander [he/him]]

(In reply to Nathan Froyd [:froydnj] from comment #2)
> My understanding is that building with clang on Android is orthogonal to
> whether the Android builds get their NDK via tooltool or the taskgraph.

But if we only build with clang, that would be only fetched via the taskgraph, solving this particular problem: we wouldn't need to produce a repack of Google's NDK at all.

Comment 4

[Nathan Froyd [:froydnj]]

(In reply to Nick Alexander :nalexander from comment #3)
> (In reply to Nathan Froyd [:froydnj] from comment #2)
> > My understanding is that building with clang on Android is orthogonal to
> > whether the Android builds get their NDK via tooltool or the taskgraph.
> 
> But if we only build with clang, that would be only fetched via the
> taskgraph, solving this particular problem: we wouldn't need to produce a
> repack of Google's NDK at all.

That's true, but we're not building with the clang from our current taskgraph, we're building with the clang out of the NDK.  We also need the NDK headers, libraries, etc. in any event.

Comment 5

[Nick Alexander :nalexander [he/him]]

(In reply to Nathan Froyd [:froydnj] from comment #4)
> (In reply to Nick Alexander :nalexander from comment #3)
> > (In reply to Nathan Froyd [:froydnj] from comment #2)
> > > My understanding is that building with clang on Android is orthogonal to
> > > whether the Android builds get their NDK via tooltool or the taskgraph.
> > 
> > But if we only build with clang, that would be only fetched via the
> > taskgraph, solving this particular problem: we wouldn't need to produce a
> > repack of Google's NDK at all.
> 
> That's true, but we're not building with the clang from our current
> taskgraph, we're building with the clang out of the NDK.  We also need the
> NDK headers, libraries, etc. in any event.

Ah!  I had forgotten that.  Thanks!

Comment 6

[Nathan Froyd [:froydnj]]

Attachment: part 1 - have a single point of truth for the NDK version

...at least in mozboot/android.py.

(This was going to be preparatory work for other patches, but I turned out not
to need the other patches.  This is a nice cleanup, though.)

Comment 7

[Nathan Froyd [:froydnj]]

Attachment: part 2 - add an --ndk-only option to mozboot/android.py

This option will be useful for our NDK repackaging task.

Comment 8

[Nathan Froyd [:froydnj]]

Attachment: part 3 - add an Android NDK repackaging task

We'd like to install the NDK through the Android SDK manager.  But we
can't pin versions of the NDK with the SDK manager, and so Google
can silently upgrade the NDK on us.  Since that is undesirable, this is
the next best thing.

With the toolchain task in hand, we can make all the relevant tasks
depend on the toolchain task and remove the download of the NDK from
tooltool as well.

I think this ought to work, but taskcluster complains at me:

[task 2018-01-30T21:55:58.421180Z] In other words you are missing scopes from one of the options:
[task 2018-01-30T21:55:58.421193Z]  * Option 0:
[task 2018-01-30T21:55:58.421214Z]     - "queue:get-artifact:project/gecko/android-ndk/*"

AFAICT, this is because I have the worker.artifacts.name property set in my
task description.  I *think* I can take it out and just have
run.toolchain-artifact, but the SDK does the worker.artifacts.name dance (as
well as run.toolchain-artifact) and I'd like to keep things consistent...  Or
do I have to have the scope set up regardless of which way I go?

Comment 9

[Nick Alexander :nalexander [he/him]]

Comment on attachment 8946857 [details] [diff] [review]
part 1 - have a single point of truth for the NDK version

Review of attachment 8946857 [details] [diff] [review]:
-----------------------------------------------------------------

Yes please!

Comment 10

[Nick Alexander :nalexander [he/him]]

Comment on attachment 8946858 [details] [diff] [review]
part 2 - add an --ndk-only option to mozboot/android.py

Review of attachment 8946858 [details] [diff] [review]:
-----------------------------------------------------------------

If this helps you, I'm happy with it.

::: python/mozboot/mozboot/android.py
@@ +300,5 @@
>  
>      options, _ = parser.parse_args(argv)
>  
> +    if options.artifact_mode and options.ndk_only:
> +        raise NotImplementedError('Use no options to install the NDK and the SDK.')

nit: this is a little awkward -- maybe, 'Cannot have --artifact-mode and --ndk-only.  To install the NDK and the SDK, omit both options.'

Comment 11

[Nick Alexander :nalexander [he/him]]

Comment on attachment 8946860 [details] [diff] [review]
part 3 - add an Android NDK repackaging task

Review of attachment 8946860 [details] [diff] [review]:
-----------------------------------------------------------------

Technically, this is fine.  Socially, this will make the toolchain artifacts private, which I think is what we want.  (That is, we're not allowed to redistribute the NDK.)  And that is what's causing your scope issue -- see https://bugzilla.mozilla.org/show_bug.cgi?id=1405408.  I think there's some magic needed to assign the scopes to builders by default, and dustin can get that done for you.  I love how straightforward this is!

Comment 12

[Dustin J. Mitchell [:dustin] (he/him)]

Comment on attachment 8946860 [details] [diff] [review]
part 3 - add an Android NDK repackaging task

Review of attachment 8946860 [details] [diff] [review]:
-----------------------------------------------------------------

This looks about right -- glandium's the authority on toolchains, though.

As for the scope, it's coming from the `toolchain-artifact` property.  Downstream consumers of that task need to download the artifact, and taskcluster/taskgraph/transforms/use_toolchains.py adds the dirname with `/*` appended to the list of scopes.

The SDK scope is given in https://tools.taskcluster.net/auth/roles/moz-tree:level:1:* -- we can do the same for the NDK scop.

Comment 13

[Dustin J. Mitchell [:dustin] (he/him)]

I changed the role to allow project/gecko/android-* which should cover both cases.

Comment 14

[Nathan Froyd [:froydnj]]

(In reply to Dustin J. Mitchell [:dustin] from comment #13)
> I changed the role to allow project/gecko/android-* which should cover both
> cases.

Thank you!

Comment 15

[Mike Hommey [:glandium]]

Comment on attachment 8946860 [details] [diff] [review]
part 3 - add an Android NDK repackaging task

Review of attachment 8946860 [details] [diff] [review]:
-----------------------------------------------------------------

::: taskcluster/scripts/misc/repack-android-ndk-linux.sh
@@ +14,5 @@
> +./mach python python/mozboot/mozboot/android.py --ndk-only --no-interactive
> +
> +# Don't generate a tarball with a versioned NDK directory.
> +mv $HOME/.mozbuild/android-ndk-* $HOME/.mozbuild/android-ndk
> +tar cf - -C /builds/worker/.mozbuild android-ndk | xz > $UPLOAD_DIR/android-ndk.tar.xz

You can use tar -Jcf instead of piping to xz.

Comment 16

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/mozilla-inbound/rev/093e449b1705
part 1 - have a single point of truth for the NDK version; r=nalexander
https://hg.mozilla.org/integration/mozilla-inbound/rev/e85bd710cc88
part 2 - add an --ndk-only option to mozboot/android.py; r=nalexander
https://hg.mozilla.org/integration/mozilla-inbound/rev/43ea8f7b4d69
part 3 - add an Android NDK repackaging task; r=dustin,nalexander; f=glandium

Comment 17

[Natalia Csoregi [:nataliaCs]]

https://hg.mozilla.org/mozilla-central/rev/093e449b1705
https://hg.mozilla.org/mozilla-central/rev/e85bd710cc88
https://hg.mozilla.org/mozilla-central/rev/43ea8f7b4d69

Comment 18

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/mozilla-central/rev/66585dad7efb
part 4 - add ndk toolchain task dependency to without-google-play-services build; r=nalexander; a=Aryx