1016529 - Updated android builders to include google play services (and android support v7 if it isn't already included)

Status: RESOLVED FIXED

(Infrastructure & Operations Graveyard :: CIDuty, task)

Description

[Wesley Johnston (:wesj)]

We're going to need these libraries and their resources for Chromecast support (bug 901803). Support for building against them is bug 1006158.

Comment 1

[Wesley Johnston (:wesj)]

To give more details, these are shipped with the Android SDK.

To download the support packages from the command line, you can use:
<android-sdk-root>/tools/android update sdk -t extra-android-support -u

The files we need are then in <android-sdk-root>/extras/android/support (we already ship v4, but its actually tough to only download just one, so we may already have v7 installed). For this we'll need files in:

<android-sdk-root>/extras/android/support/v7/appcompat/lib
<android-sdk-root>/extras/android/support/v7/appcompat/res
<android-sdk-root>/extras/android/support/v7/mediarouter/lib
<android-sdk-root>/extras/android/support/v7/mediarouter/res

To install play services
<android-sdk-root>/tools/android update sdk -t addon-google_apis_x86-google-19 -u
(note there's a version number in there. I'm not sure what version we need actually, but 19 is the latest I see. That also downloads the x86 tools, so we may have this installed already as well... do we?)

the files we need will be in:
<android-sdk-root>/extras/google/google_play_services/libproject/google-play-services_lib/lib
<android-sdk-root>/extras/google/google_play_services/libproject/google-play-services_lib/res

The patches in the tree depend on those being in the right place in the sdk, so they'll need to appear in those locations as well.

Comment 2

[Chris Cooper [:coop] (he/him)]

(In reply to Wesley Johnston (:wesj) from comment #1)
> To give more details, these are shipped with the Android SDK.
> 
> To download the support packages from the command line, you can use:
> <android-sdk-root>/tools/android update sdk -t extra-android-support -u
> 
> The files we need are then in <android-sdk-root>/extras/android/support (we
> already ship v4, but its actually tough to only download just one, so we may
> already have v7 installed). For this we'll need files in:
> 
> <android-sdk-root>/extras/android/support/v7/appcompat/lib
> <android-sdk-root>/extras/android/support/v7/appcompat/res
> <android-sdk-root>/extras/android/support/v7/mediarouter/lib
> <android-sdk-root>/extras/android/support/v7/mediarouter/res

These appear to be available already in the existing sdk (see below).
 
> To install play services
> <android-sdk-root>/tools/android update sdk -t
> addon-google_apis_x86-google-19 -u
> (note there's a version number in there. I'm not sure what version we need
> actually, but 19 is the latest I see. That also downloads the x86 tools, so
> we may have this installed already as well... do we?)
> 
> the files we need will be in:
> <android-sdk-root>/extras/google/google_play_services/libproject/google-play-
> services_lib/lib
> <android-sdk-root>/extras/google/google_play_services/libproject/google-play-
> services_lib/res
> 
> The patches in the tree depend on those being in the right place in the sdk,
> so they'll need to appear in those locations as well.

We use tooltool for the Android sdk deployments now which has the added benefit of you being able to affect change here yourself. By updating the sdk package to include the google_play_services and the in-tree manifest, you can iterate on this yourself.

tbsaunde recently deployed a similar change over in bug 933189.

Tooltool docs are here: https://wiki.mozilla.org/ReleaseEngineering/Applications/Tooltool

Comment 3

[Wesley Johnston (:wesj)]

Attachment: Patch

I'm uploading the SDK (an rpm scraped from various instructions I've found?) to

http://people.mozilla.org/~wjohnston/android-sdk.tar.xz

This updates the tooltool manifest files.

I used 19 because I don't know why we wouldn't use 19, and I don't really trust the android tools to pull older version, but I can try to pull 17 if we'd rather stay there.

This feels a lot like voodoo magic to me. Hope its right.

Comment 4

[Mike Hommey [:glandium]]

Comment on attachment 8451936 [details] [diff] [review]
Patch

Review of attachment 8451936 [details] [diff] [review]:
-----------------------------------------------------------------

::: mobile/android/config/tooltool-manifests/android-x86/releng.manifest
@@ +5,5 @@
>  "algorithm": "sha512",
>  "filename": "android-ndk.tar.bz2"
>  },
>  {
> +"size": 1193500315,

1.1GB to replace a 156MB archive? That doesn't sound right.

Comment 5

[Mike Hommey [:glandium]]

Trevor, did you have a script to create the android sdk tarball? Bug 859984 only has one for the NDK.

Comment 6

[Trevor Saunders (:tbsaunde)]

(In reply to Mike Hommey [:glandium] from comment #5)
> Trevor, did you have a script to create the android sdk tarball? Bug 859984
> only has one for the NDK.

embarrassing, but I don't think I ever wrote one, certainly the last time I did this I did it by hand more or less following the releng wiki page on making sdk  rpm's as for what to remove from the sdk.

Comment 7

[Trevor Saunders (:tbsaunde)]

fwiw I believe this is the wiki page I got the list of what to remove from https://wiki.mozilla.org/ReleaseEngineering/How_To/Build_An_Android_SDK_rpm

Comment 8

[Wesley Johnston (:wesj)]

That's what I went with. Something must have gone awry during my rpm build. This seems better:

http://people.mozilla.org/~wjohnston/android_sdk_r19.tgz

Comment 9

[Wesley Johnston (:wesj)]

That's a bit bigger than the last set. Some of its going from sdk 19.0.2 to 19.1, but a bigger chunk seems to come from monitor. We could probably rip that out if its helpful...

Looking at Trevor's stuff, I guess I didn't need to do the rpm bit either.

Comment 10

[Wesley Johnston (:wesj)]

Attachment: Patch

I'm still not really sure if you guys need an rpm or just a gzippped-tar of the modified sdk. The instructions talk about rpmbuild. The last upload that I can see just used a zipped up tar.

Comment 11

[Chris Cooper [:coop] (he/him)]

(In reply to Wesley Johnston (:wesj) from comment #10)
> Created attachment 8455404 [details] [diff] [review]
> Patch
> 
> I'm still not really sure if you guys need an rpm or just a gzippped-tar of
> the modified sdk. The instructions talk about rpmbuild. The last upload that
> I can see just used a zipped up tar.

Wes: according to the manifest, we need a tarball, specifically android-sdk.tar.xz. 

Based on the location of your rpm above, is http://people.mozilla.org/~wjohnston/android-sdk.tar.xz the correct file? I ask because the sha512sum doesn't seem to match what's in your manifest patch.

cc-ing Simone who can help deploy to tooltool once we have the correct file.

Comment 12

[Mike Hommey [:glandium]]

Comment on attachment 8455404 [details] [diff] [review]
Patch

Review of attachment 8455404 [details] [diff] [review]:
-----------------------------------------------------------------

cf. comment 11.

Comment 13

[Trevor Saunders (:tbsaunde)]

do you still need something from? I'm not really sure what you want to know at this point.

Comment 14

[Wesley Johnston (:wesj)]

http://people.mozilla.org/~wjohnston/android-sdk.tar.xz

Coop does this build look better? I dropped the rpm stuff since it wasn't included int he previous sdk stuff.

Comment 15

[Simone Bruno [:simone]]

hi Wesley, in order to use a new version of android-sdk.tar.xz, two steps are needed:

1) upload the new version of android-sdk.tar.xz to tooltool
2) change intree tooltool manifests (usually .tt files or releng.manifest files) so that that new version is actually used in builds.

In the past, uploads to tooltool were requested to and managed by releng. Currently any authorized user can run tooltool uploads. I raised Bug 1039994 to enable you to tooltool uploads. Once that's done, you can run tooltool uploads on your own by following instructions here: https://wiki.mozilla.org/ReleaseEngineering/Applications/Tooltool#How_to_upload_to_tooltool.
You just need to make sure the correct artifact is uploaded: while uploading, tooltool itself will tell you the sha512 checksum and size to be used in tooltool manifest patches.

Please let me know in case of any issues or questions, I'll be happy to help!

Comment 16

[Simone Bruno [:simone]]

Bug 1039994 is closed; :wesj, you should be able to upload files to tooltool following the instructions provided in the link in previous comment.

Comment 17

[Chris Cooper [:coop] (he/him)]

(In reply to Wesley Johnston (:wesj) from comment #14)
> http://people.mozilla.org/~wjohnston/android-sdk.tar.xz
> 
> Coop does this build look better? I dropped the rpm stuff since it wasn't
> included int he previous sdk stuff.

Hrmm, I'm not sure. The suffix is .xz, but it seems to actually be a gzip archive. Is that intentional?

Comment 18

[Wesley Johnston (:wesj)]

(In reply to Chris Cooper [:coop] from comment #17)
> Hrmm, I'm not sure. The suffix is .xz, but it seems to actually be a gzip
> archive. Is that intentional?

Nope. I just only ever use three tar commands. Updated it with xz compression.

Comment 19

[Wesley Johnston (:wesj)]

This feature is supposed to ship in 33, but waiting for this seems likely to make it slip at this point. Looking back, my trying to help seems to be delaying things more than helping. Can I get any help from someone who knows their way through this?

Comment 20

[Trevor Saunders (:tbsaunde)]

people.mozilla.org/~tsaunders/android-sdk.tar.xz should be what you need. I tested it builds with a fairly recent tree, but it wouldn't hurt to test your patches that need this work.  You'll also need to update build/autoconf/android.m4 to look for zipalign in ANDROID_BUILD_TOOLS because google loves to rearrange the sdk for some reason.
I don't have the bits to do tooltool uploads so you'll have to get someone else to upload that file.  THen you just have to update the manifests.
The sdk has grown about 70mb since the last time, I think mostly because of the stuff you need, but I didn't look into it too much.

Comment 21

[Chris Cooper [:coop] (he/him)]

Wes: can you try the instructions for uploading yourself that Simone linked in comment #15? If there are problems, Simone can help further with the upload.

Comment 22

[Wesley Johnston (:wesj)]

Attachment: Patch

I think I uploaded this. I can see a file with this hash on the server at least :) Thanks for the help coop.

Comment 23

[Chris Cooper [:coop] (he/him)]

Comment on attachment 8460290 [details] [diff] [review]
Patch

Review of attachment 8460290 [details] [diff] [review]:
-----------------------------------------------------------------

I see the new hash on the tooltool server, so LGTM.

Comment 24

[Simone Bruno [:simone]]

Wesley: I think the patch you uploaded removes a couple of commas which make the tooltool manifests invalid, be careful of lines:

>>> -},
>>> +}

in the patch, since the comma is required as separator for different files sections.

Furthermore, I noticed that you tried three uploads with three different files:

[
{
"size": 226882001,
"digest": "8c4278c5a0639b9af0c0ec2e10e6a4e140781685d24a9bb9e097e036d2190f376886bd0ecc1dae607c38659933b2e66240f81a979197b2f7088eb2d485b44232",
"algorithm": "sha512",
"filename": "android-sdk.tar.xz"
}
]
[
{
"size": 201739236,
"digest": "2c0b1c236b3c72e419c84aaee72e086d6f863c8b33dcb2806b9472ef5827968a35894f49a556b266f3aefd8529df3cdaa6cc30e7c2cf574a9230c4525ca839fe",
"algorithm": "sha512",
"filename": "android-sdk.tar.xz"
}
]
[
{
"size": 190380996,
"digest": "9e2da00cd4783e10a52c144d2dc9de71b9ab9772f26c9a4e36b9a744e05edd1d13c212b976a3d2f6cbcb86bce00b78fd262b47f8573eff2ab9db0ed2d4cfb8c5",
"algorithm": "sha512",
"filename": "android-sdk.tar.xz"
}
]

In the patched manifests you use only one of them - can I remove the other two from the tooltool server to cleanup?

Comment 25

[Wesley Johnston (:wesj)]

Yeah, I caught that when I've been playing on try. The first tar I uploaded had things in an android-sdk-v19 subfolder that broke builds. The next only had the v19 sdk in it, but we're still on v17 (I figured we'd update later). The latest has problems with zipalign not being found. Looking into that. If you want to remove them all feel free. I don't know why I'm not testing locally first :(

Comment 26

[Wesley Johnston (:wesj)]

Whoops. Hold on to the 9e2da00cd4783e10a52c144d2dc9de71b9ab9772f26c9a4e36b9a744e05edd1d13c212b976a3d2f6cbcb86bce00b78fd262b47f8573eff2ab9db0ed2d4cfb8c5 build. Just needed to pull in some android.m4 changes and it works fine. Thanks :)

Comment 27

[Wesley Johnston (:wesj)]

Attachment: Patch

Sorry for the delay. This looks good:
https://tbpl.mozilla.org/?tree=Try&rev=6de6357d1524

push with casting enabled worked locally, but waiting on it now:
https://tbpl.mozilla.org/?tree=Try&rev=7b9d31258f9b

Comment 28

[Wesley Johnston (:wesj)]

Looks like the Android builders use jdk1.6. Play services is pre-built using jdk1.7. The two are not getting along. Pushed again with -Werror removed to see what comes out. We can't really fix play services. Can we upgrade the builders to jdk1.7?

Comment 29

[Wesley Johnston (:wesj)]

And even disable warnings gives errors in the annotation processor:

Exception in thread "main" java.lang.UnsupportedClassVersionError: com/google/android/gms/cast/CastDevice : Unsupported major.minor version 51.0

Comment 30

[Simone Bruno [:simone]]

Comment on attachment 8461034 [details] [diff] [review]
Patch

Review of attachment 8461034 [details] [diff] [review]:
-----------------------------------------------------------------

9e2da00cd4783e10a52c144d2dc9de71b9ab9772f26c9a4e36b9a744e05edd1d13c212b976a3d2f6cbcb86bce00b78fd262b47f8573eff2ab9db0ed2d4cfb8c5 has been uploaded, so formally this patch looks ok.

Comment 31

[Wesley Johnston (:wesj)]

Long as we don't enable chromecast (on the builders), this is a no-op change basically. Landed:
https://hg.mozilla.org/integration/fx-team/rev/7086cfb5b6fe

Comment 32

[Ed Morley [:emorley]]

Backed out for causing:
https://tbpl.mozilla.org/php/getParsedLog.php?id=44522466&tree=Fx-Team
/builds/slave/fx-team-and-000000000000000000/build/android-sdk-linux/build-tools/19.0.2/aapt package -f -m -M AndroidManifest.xml -I /builds/slave/fx-team-and-000000000000000000/build/android-sdk-linux/platforms/android-17/android.jar --auto-add-overlay -S /builds/slave/fx-team-and-000000000000000000/build/mobile/android/base/crashreporter/res -S /builds/slave/fx-team-and-000000000000000000/build/mobile/android/base/resources -S /builds/slave/fx-team-and-000000000000000000/build/mobile/android/branding/nightly/res -S /builds/slave/fx-team-and-000000000000000000/build/obj-firefox/mobile/android/base/res   --custom-package org.mozilla.gecko --non-constant-id -F gecko.ap_ -J generated/ --output-text-symbols ./ --ignore-assets "!.svn:!.git:.*:<dir>_*:!CVS:!thumbs.db:!picasa.ini:!*.scc:*~:#*:*.rej:*.orig"
/bin/sh: /builds/slave/fx-team-and-000000000000000000/build/android-sdk-linux/build-tools/19.0.2/aapt: No such file or directory
make[5]: *** [.aapt.deps] Error 127

remote:   https://hg.mozilla.org/integration/fx-team/rev/72cdf2155ca6

Comment 33

[Wesley Johnston (:wesj)]

I think maybe I just need to force these to clobber. i.e. I don't see configure running on the failing builds, so they're trying to use non-existent paths. Since try does that automatically, I'm not sure how to test. Can someone give me a thumbs up for that before I reland?

Comment 34

[Wesley Johnston (:wesj)]

Releng agreed a clobber should fix this. Lets try again!:
https://hg.mozilla.org/integration/fx-team/rev/d3e90a9d7ef6

Comment 35

[Wes Kocher (:KWierso) (Not reading bugmail; email directly if needed)]

https://hg.mozilla.org/mozilla-central/rev/d3e90a9d7ef6

Comment 36

[Richard Newman [:rnewman]]

For the benefit of those performing future forensics, this change seems to have bumped our APK size by about 100KB.

https://hg.mozilla.org/integration/mozilla-inbound/rev/b0a916157c9a		29036688
https://hg.mozilla.org/integration/mozilla-inbound/rev/de374e83ed83		29138304

Comment 37

[Richard Newman [:rnewman]]

Bad sleuthing on my part. Wes found the real culprit: Bug 1036299.

Comment 38

[Wesley Johnston (:wesj)]

Comment on attachment 8461034 [details] [diff] [review]
Patch

Approval Request Comment
[Feature/regressing bug #]: new feature
[User impact if declined]: Can't enable sending video to chromecast
[Describe test coverage new/current, TBPL]: None :(
[Risks and why]: This is pretty low risk. Just enabling a slightly different sdk package for aurora that is the same sdk as before, but now includes google play services as well. That code isn't even bundled with fennec until bug 1042238 (already approved for uplift, but dependent on this).
[String/UUID change made/needed]: none

Comment 39

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/releases/mozilla-aurora/rev/83411dedecb4