Updated android builders to include google play services (and android support v7 if it isn't already included)

RESOLVED FIXED

Status

Release Engineering
Platform Support
RESOLVED FIXED
3 years ago
3 years ago

People

(Reporter: wesj, Assigned: wesj)

Tracking

unspecified
x86_64
Linux
Dependency tree / graph

Firefox Tracking Flags

(firefox33 fixed, firefox34 fixed)

Details

Attachments

(2 attachments, 2 obsolete attachments)

(Assignee)

Description

3 years ago
We're going to need these libraries and their resources for Chromecast support (bug 901803). Support for building against them is bug 1006158.
(Assignee)

Updated

3 years ago
Blocks: 901803
(Assignee)

Comment 1

3 years ago
To give more details, these are shipped with the Android SDK.

To download the support packages from the command line, you can use:
<android-sdk-root>/tools/android update sdk -t extra-android-support -u

The files we need are then in <android-sdk-root>/extras/android/support (we already ship v4, but its actually tough to only download just one, so we may already have v7 installed). For this we'll need files in:

<android-sdk-root>/extras/android/support/v7/appcompat/lib
<android-sdk-root>/extras/android/support/v7/appcompat/res
<android-sdk-root>/extras/android/support/v7/mediarouter/lib
<android-sdk-root>/extras/android/support/v7/mediarouter/res

To install play services
<android-sdk-root>/tools/android update sdk -t addon-google_apis_x86-google-19 -u
(note there's a version number in there. I'm not sure what version we need actually, but 19 is the latest I see. That also downloads the x86 tools, so we may have this installed already as well... do we?)

the files we need will be in:
<android-sdk-root>/extras/google/google_play_services/libproject/google-play-services_lib/lib
<android-sdk-root>/extras/google/google_play_services/libproject/google-play-services_lib/res

The patches in the tree depend on those being in the right place in the sdk, so they'll need to appear in those locations as well.
Flags: needinfo?(coop)

Comment 2

3 years ago
(In reply to Wesley Johnston (:wesj) from comment #1)
> To give more details, these are shipped with the Android SDK.
> 
> To download the support packages from the command line, you can use:
> <android-sdk-root>/tools/android update sdk -t extra-android-support -u
> 
> The files we need are then in <android-sdk-root>/extras/android/support (we
> already ship v4, but its actually tough to only download just one, so we may
> already have v7 installed). For this we'll need files in:
> 
> <android-sdk-root>/extras/android/support/v7/appcompat/lib
> <android-sdk-root>/extras/android/support/v7/appcompat/res
> <android-sdk-root>/extras/android/support/v7/mediarouter/lib
> <android-sdk-root>/extras/android/support/v7/mediarouter/res

These appear to be available already in the existing sdk (see below).
 
> To install play services
> <android-sdk-root>/tools/android update sdk -t
> addon-google_apis_x86-google-19 -u
> (note there's a version number in there. I'm not sure what version we need
> actually, but 19 is the latest I see. That also downloads the x86 tools, so
> we may have this installed already as well... do we?)
> 
> the files we need will be in:
> <android-sdk-root>/extras/google/google_play_services/libproject/google-play-
> services_lib/lib
> <android-sdk-root>/extras/google/google_play_services/libproject/google-play-
> services_lib/res
> 
> The patches in the tree depend on those being in the right place in the sdk,
> so they'll need to appear in those locations as well.

We use tooltool for the Android sdk deployments now which has the added benefit of you being able to affect change here yourself. By updating the sdk package to include the google_play_services and the in-tree manifest, you can iterate on this yourself.

tbsaunde recently deployed a similar change over in bug 933189.

Tooltool docs are here: https://wiki.mozilla.org/ReleaseEngineering/Applications/Tooltool
Flags: needinfo?(coop)
(Assignee)

Comment 3

3 years ago
Created attachment 8451936 [details] [diff] [review]
Patch

I'm uploading the SDK (an rpm scraped from various instructions I've found?) to

http://people.mozilla.org/~wjohnston/android-sdk.tar.xz

This updates the tooltool manifest files.

I used 19 because I don't know why we wouldn't use 19, and I don't really trust the android tools to pull older version, but I can try to pull 17 if we'd rather stay there.

This feels a lot like voodoo magic to me. Hope its right.
Attachment #8451936 - Flags: review?(mh+mozilla)
Comment on attachment 8451936 [details] [diff] [review]
Patch

Review of attachment 8451936 [details] [diff] [review]:
-----------------------------------------------------------------

::: mobile/android/config/tooltool-manifests/android-x86/releng.manifest
@@ +5,5 @@
>  "algorithm": "sha512",
>  "filename": "android-ndk.tar.bz2"
>  },
>  {
> +"size": 1193500315,

1.1GB to replace a 156MB archive? That doesn't sound right.
Attachment #8451936 - Flags: review?(mh+mozilla) → review-
Trevor, did you have a script to create the android sdk tarball? Bug 859984 only has one for the NDK.
Flags: needinfo?(trev.saunders)
(In reply to Mike Hommey [:glandium] from comment #5)
> Trevor, did you have a script to create the android sdk tarball? Bug 859984
> only has one for the NDK.

embarrassing, but I don't think I ever wrote one, certainly the last time I did this I did it by hand more or less following the releng wiki page on making sdk  rpm's as for what to remove from the sdk.
Flags: needinfo?(trev.saunders)
fwiw I believe this is the wiki page I got the list of what to remove from https://wiki.mozilla.org/ReleaseEngineering/How_To/Build_An_Android_SDK_rpm
(Assignee)

Comment 8

3 years ago
That's what I went with. Something must have gone awry during my rpm build. This seems better:

http://people.mozilla.org/~wjohnston/android_sdk_r19.tgz
(Assignee)

Comment 9

3 years ago
That's a bit bigger than the last set. Some of its going from sdk 19.0.2 to 19.1, but a bigger chunk seems to come from monitor. We could probably rip that out if its helpful...

Looking at Trevor's stuff, I guess I didn't need to do the rpm bit either.
(Assignee)

Comment 10

3 years ago
Created attachment 8455404 [details] [diff] [review]
Patch

I'm still not really sure if you guys need an rpm or just a gzippped-tar of the modified sdk. The instructions talk about rpmbuild. The last upload that I can see just used a zipped up tar.
Attachment #8451936 - Attachment is obsolete: true
Attachment #8455404 - Flags: review?(mh+mozilla)
(Assignee)

Updated

3 years ago
Flags: needinfo?(trev.saunders)
(In reply to Wesley Johnston (:wesj) from comment #10)
> Created attachment 8455404 [details] [diff] [review]
> Patch
> 
> I'm still not really sure if you guys need an rpm or just a gzippped-tar of
> the modified sdk. The instructions talk about rpmbuild. The last upload that
> I can see just used a zipped up tar.

Wes: according to the manifest, we need a tarball, specifically android-sdk.tar.xz. 

Based on the location of your rpm above, is http://people.mozilla.org/~wjohnston/android-sdk.tar.xz the correct file? I ask because the sha512sum doesn't seem to match what's in your manifest patch.

cc-ing Simone who can help deploy to tooltool once we have the correct file.
Comment on attachment 8455404 [details] [diff] [review]
Patch

Review of attachment 8455404 [details] [diff] [review]:
-----------------------------------------------------------------

cf. comment 11.
Attachment #8455404 - Flags: review?(mh+mozilla) → review-
do you still need something from? I'm not really sure what you want to know at this point.
Flags: needinfo?(trev.saunders)
(Assignee)

Comment 14

3 years ago
http://people.mozilla.org/~wjohnston/android-sdk.tar.xz

Coop does this build look better? I dropped the rpm stuff since it wasn't included int he previous sdk stuff.
Flags: needinfo?(coop)
hi Wesley, in order to use a new version of android-sdk.tar.xz, two steps are needed:

1) upload the new version of android-sdk.tar.xz to tooltool
2) change intree tooltool manifests (usually .tt files or releng.manifest files) so that that new version is actually used in builds.

In the past, uploads to tooltool were requested to and managed by releng. Currently any authorized user can run tooltool uploads. I raised Bug 1039994 to enable you to tooltool uploads. Once that's done, you can run tooltool uploads on your own by following instructions here: https://wiki.mozilla.org/ReleaseEngineering/Applications/Tooltool#How_to_upload_to_tooltool.
You just need to make sure the correct artifact is uploaded: while uploading, tooltool itself will tell you the sha512 checksum and size to be used in tooltool manifest patches.

Please let me know in case of any issues or questions, I'll be happy to help!
Bug 1039994 is closed; :wesj, you should be able to upload files to tooltool following the instructions provided in the link in previous comment.
(In reply to Wesley Johnston (:wesj) from comment #14)
> http://people.mozilla.org/~wjohnston/android-sdk.tar.xz
> 
> Coop does this build look better? I dropped the rpm stuff since it wasn't
> included int he previous sdk stuff.

Hrmm, I'm not sure. The suffix is .xz, but it seems to actually be a gzip archive. Is that intentional?
Flags: needinfo?(coop)
(Assignee)

Comment 18

3 years ago
(In reply to Chris Cooper [:coop] from comment #17)
> Hrmm, I'm not sure. The suffix is .xz, but it seems to actually be a gzip
> archive. Is that intentional?

Nope. I just only ever use three tar commands. Updated it with xz compression.
Flags: needinfo?(coop)
(Assignee)

Comment 19

3 years ago
This feature is supposed to ship in 33, but waiting for this seems likely to make it slip at this point. Looking back, my trying to help seems to be delaying things more than helping. Can I get any help from someone who knows their way through this?
Flags: needinfo?(trev.saunders)
people.mozilla.org/~tsaunders/android-sdk.tar.xz should be what you need. I tested it builds with a fairly recent tree, but it wouldn't hurt to test your patches that need this work.  You'll also need to update build/autoconf/android.m4 to look for zipalign in ANDROID_BUILD_TOOLS because google loves to rearrange the sdk for some reason.
I don't have the bits to do tooltool uploads so you'll have to get someone else to upload that file.  THen you just have to update the manifests.
The sdk has grown about 70mb since the last time, I think mostly because of the stuff you need, but I didn't look into it too much.
Flags: needinfo?(trev.saunders)
Wes: can you try the instructions for uploading yourself that Simone linked in comment #15? If there are problems, Simone can help further with the upload.
Flags: needinfo?(coop)
(Assignee)

Comment 22

3 years ago
Created attachment 8460290 [details] [diff] [review]
Patch

I think I uploaded this. I can see a file with this hash on the server at least :) Thanks for the help coop.
Attachment #8455404 - Attachment is obsolete: true
Attachment #8460290 - Flags: review?(coop)
Comment on attachment 8460290 [details] [diff] [review]
Patch

Review of attachment 8460290 [details] [diff] [review]:
-----------------------------------------------------------------

I see the new hash on the tooltool server, so LGTM.
Attachment #8460290 - Flags: review?(coop) → review+
Wesley: I think the patch you uploaded removes a couple of commas which make the tooltool manifests invalid, be careful of lines:

>>> -},
>>> +}

in the patch, since the comma is required as separator for different files sections.

Furthermore, I noticed that you tried three uploads with three different files:

[
{
"size": 226882001,
"digest": "8c4278c5a0639b9af0c0ec2e10e6a4e140781685d24a9bb9e097e036d2190f376886bd0ecc1dae607c38659933b2e66240f81a979197b2f7088eb2d485b44232",
"algorithm": "sha512",
"filename": "android-sdk.tar.xz"
}
]
[
{
"size": 201739236,
"digest": "2c0b1c236b3c72e419c84aaee72e086d6f863c8b33dcb2806b9472ef5827968a35894f49a556b266f3aefd8529df3cdaa6cc30e7c2cf574a9230c4525ca839fe",
"algorithm": "sha512",
"filename": "android-sdk.tar.xz"
}
]
[
{
"size": 190380996,
"digest": "9e2da00cd4783e10a52c144d2dc9de71b9ab9772f26c9a4e36b9a744e05edd1d13c212b976a3d2f6cbcb86bce00b78fd262b47f8573eff2ab9db0ed2d4cfb8c5",
"algorithm": "sha512",
"filename": "android-sdk.tar.xz"
}
]

In the patched manifests you use only one of them - can I remove the other two from the tooltool server to cleanup?
Flags: needinfo?(wjohnston)
(Assignee)

Comment 25

3 years ago
Yeah, I caught that when I've been playing on try. The first tar I uploaded had things in an android-sdk-v19 subfolder that broke builds. The next only had the v19 sdk in it, but we're still on v17 (I figured we'd update later). The latest has problems with zipalign not being found. Looking into that. If you want to remove them all feel free. I don't know why I'm not testing locally first :(
Flags: needinfo?(wjohnston)
(Assignee)

Comment 26

3 years ago
Whoops. Hold on to the 9e2da00cd4783e10a52c144d2dc9de71b9ab9772f26c9a4e36b9a744e05edd1d13c212b976a3d2f6cbcb86bce00b78fd262b47f8573eff2ab9db0ed2d4cfb8c5 build. Just needed to pull in some android.m4 changes and it works fine. Thanks :)
(Assignee)

Comment 27

3 years ago
Created attachment 8461034 [details] [diff] [review]
Patch

Sorry for the delay. This looks good:
https://tbpl.mozilla.org/?tree=Try&rev=6de6357d1524

push with casting enabled worked locally, but waiting on it now:
https://tbpl.mozilla.org/?tree=Try&rev=7b9d31258f9b
Attachment #8461034 - Flags: review?(sbruno)
(Assignee)

Comment 28

3 years ago
Looks like the Android builders use jdk1.6. Play services is pre-built using jdk1.7. The two are not getting along. Pushed again with -Werror removed to see what comes out. We can't really fix play services. Can we upgrade the builders to jdk1.7?
(Assignee)

Comment 29

3 years ago
And even disable warnings gives errors in the annotation processor:

Exception in thread "main" java.lang.UnsupportedClassVersionError: com/google/android/gms/cast/CastDevice : Unsupported major.minor version 51.0
Comment on attachment 8461034 [details] [diff] [review]
Patch

Review of attachment 8461034 [details] [diff] [review]:
-----------------------------------------------------------------

9e2da00cd4783e10a52c144d2dc9de71b9ab9772f26c9a4e36b9a744e05edd1d13c212b976a3d2f6cbcb86bce00b78fd262b47f8573eff2ab9db0ed2d4cfb8c5 has been uploaded, so formally this patch looks ok.
Attachment #8461034 - Flags: review?(sbruno) → review+
(Assignee)

Comment 31

3 years ago
Long as we don't enable chromecast (on the builders), this is a no-op change basically. Landed:
https://hg.mozilla.org/integration/fx-team/rev/7086cfb5b6fe
Backed out for causing:
https://tbpl.mozilla.org/php/getParsedLog.php?id=44522466&tree=Fx-Team
/builds/slave/fx-team-and-000000000000000000/build/android-sdk-linux/build-tools/19.0.2/aapt package -f -m -M AndroidManifest.xml -I /builds/slave/fx-team-and-000000000000000000/build/android-sdk-linux/platforms/android-17/android.jar --auto-add-overlay -S /builds/slave/fx-team-and-000000000000000000/build/mobile/android/base/crashreporter/res -S /builds/slave/fx-team-and-000000000000000000/build/mobile/android/base/resources -S /builds/slave/fx-team-and-000000000000000000/build/mobile/android/branding/nightly/res -S /builds/slave/fx-team-and-000000000000000000/build/obj-firefox/mobile/android/base/res   --custom-package org.mozilla.gecko --non-constant-id -F gecko.ap_ -J generated/ --output-text-symbols ./ --ignore-assets "!.svn:!.git:.*:<dir>_*:!CVS:!thumbs.db:!picasa.ini:!*.scc:*~:#*:*.rej:*.orig"
/bin/sh: /builds/slave/fx-team-and-000000000000000000/build/android-sdk-linux/build-tools/19.0.2/aapt: No such file or directory
make[5]: *** [.aapt.deps] Error 127

remote:   https://hg.mozilla.org/integration/fx-team/rev/72cdf2155ca6
(Assignee)

Comment 33

3 years ago
I think maybe I just need to force these to clobber. i.e. I don't see configure running on the failing builds, so they're trying to use non-existent paths. Since try does that automatically, I'm not sure how to test. Can someone give me a thumbs up for that before I reland?
(Assignee)

Comment 34

3 years ago
Releng agreed a clobber should fix this. Lets try again!:
https://hg.mozilla.org/integration/fx-team/rev/d3e90a9d7ef6
https://hg.mozilla.org/mozilla-central/rev/d3e90a9d7ef6
Assignee: nobody → wjohnston
Status: NEW → RESOLVED
Last Resolved: 3 years ago
Resolution: --- → FIXED
For the benefit of those performing future forensics, this change seems to have bumped our APK size by about 100KB.

https://hg.mozilla.org/integration/mozilla-inbound/rev/b0a916157c9a		29036688
https://hg.mozilla.org/integration/mozilla-inbound/rev/de374e83ed83		29138304
Bad sleuthing on my part. Wes found the real culprit: Bug 1036299.
Blocks: 1042238
(Assignee)

Comment 38

3 years ago
Comment on attachment 8461034 [details] [diff] [review]
Patch

Approval Request Comment
[Feature/regressing bug #]: new feature
[User impact if declined]: Can't enable sending video to chromecast
[Describe test coverage new/current, TBPL]: None :(
[Risks and why]: This is pretty low risk. Just enabling a slightly different sdk package for aurora that is the same sdk as before, but now includes google play services as well. That code isn't even bundled with fennec until bug 1042238 (already approved for uplift, but dependent on this).
[String/UUID change made/needed]: none
Attachment #8461034 - Flags: approval-mozilla-aurora?
status-firefox33: --- → affected
status-firefox34: --- → fixed
Attachment #8461034 - Flags: approval-mozilla-aurora? → approval-mozilla-aurora+
https://hg.mozilla.org/releases/mozilla-aurora/rev/83411dedecb4
status-firefox33: affected → fixed
You need to log in before you can comment on or make changes to this bug.