909385 - Consider using new Android x86 images from android-18 platform

Status: RESOLVED WONTFIX

(Testing :: General, defect)

Description

[Geoff Brown [:gbrown]]

An android sdk update was recently released and now includes x86 emulator images for android-18. We have been using the android-17 images for all of our x86 testing up until now.

I hope that android-18 will resolve the issue described in bug 892118.

It is also possible that the emulator will be more stable (we have had problems launching more than one emulator at the same time, and armenzg has reported problems even with staggered launching, with emulators failing to start or becoming unresponsive.)

Comment 1

[Armen [:armenzg]]

I think we have been using android-18:
https://bugzilla.mozilla.org/show_bug.cgi?id=895186#c21

Comment 2

[Geoff Brown [:gbrown]]

I thought you were using the images in test-avds-aug6.tgz, from https://bugzilla.mozilla.org/show_bug.cgi?id=895186#c29 -- those are from android-17.

Comment 3

[Armen [:armenzg]]

I tried to determine which one you were using in https://bugzilla.mozilla.org/show_bug.cgi?id=895186#c21 which SDK we were using as the newer one was out.

Should we go back and try to use android-17? or can you generate the avds with android-18?

Comment 4

[Armen [:armenzg]]

needinfo wrt to comment 3. Thanks!

Comment 5

[Geoff Brown [:gbrown]]

I put together new avd definitions and images based on the android-18 x86 system images and ran all the tests. The new images do resolve the issue in bug 892118. I have not had any problems with emulator startup...but it's too small of a sample size to say anything conclusive about emulator stability yet. Mochitests and robocop run the same as with android-17, with one bonus: In android-18, the stock browser can be opened in the emulator, providing a likely solution to bug 900664. 

Unfortunately, I am seeing a lot of reftest failures. I may have made a mistake in the screen characteristics...I need to resolve this before endorsing the android-18 images.

Comment 6

[Geoff Brown [:gbrown]]

The android-18 images are based on Android 4.3, introducing restrictions on the capabilities of su (http://www.androidcentral.com/android-43-and-beyond-root-going-away-stock-roms). The only related problem for the x86 emulator tests seems to be that xpcshell tests cannot create /data/local/xpcb. I might be able to work around that.

Comment 7

[Geoff Brown [:gbrown]]

Another consequence of the su restrictions on 4.3: If an anr is reported, /data/anr/traces.txt will be reported, but the file will not be deleted, so the traces will be reported by the next test (and the next...), even if they do not generate an anr.

Comment 8

[Geoff Brown [:gbrown]]

(In reply to Geoff Brown [:gbrown] from comment #5)
> Unfortunately, I am seeing a lot of reftest failures. I may have made a
> mistake in the screen characteristics...I need to resolve this before
> endorsing the android-18 images.

The reftest failures are independent of the android-18 image. I reverted to the android-17 image (test-avds-aug6.tgz) and the reftest failures persisted.

Comment 9

[Geoff Brown [:gbrown]]

(In reply to Armen Zambrano [:armenzg] (Use needinfo flag) (Release Enginerring) (EDT/UTC-4) from comment #3)

The android-18 x86 images are superior to the android-17 images in that:
 - logcat is much tidier on 18; 17 spews a lot of useless messages
 - we can run the stock browser, enabling robocop testImportFromAndroid

Test reliability and emulator stability seems to be about the same on 17 and 18.

However, because the android-18 images are based on Android 4.3, sutagent has problems executing privileged operations. 

> Should we go back and try to use android-17? or can you generate the avds
> with android-18?

Because of the restrictions on privileged operations, we cannot use android-18 system images at this time -- stay with 17. http://people.mozilla.org/~gbrown/test-avds-aug6.tgz has the android-17 images.

It should be fine to use either the android-17 or android-18 emulator. I have a slight preference for 18, since the emulator typically improves a little with each release (but I don't actually know of any difference).

Comment 10

[Armen [:armenzg]]

Hi gbrown, where can we get the android-17 sdk?

I can't easily find an "archives" link in here:
https://developer.android.com/sdk/index.html

I'm not sure if this is where we find it or not:
http://tools.android.com/download/adt-17-preview

Comment 11

[Geoff Brown [:gbrown]]

As far as I know, the android-17 sdk download is no longer available. However, you can download the current Android SDK from http://developer.android.com/sdk/index.html and then use the SDK Manager to cherry-pick versioned components.

http://developer.android.com/tools/help/sdk-manager.html

Comment 12

[Justin Wood (:Callek)]

[18:40:46]	Callek	gbrown: to be clear, the sdk-18 is ok to *run* the images with, right?
[18:41:03]	Callek	gbrown: as in, we have no need to get/install sdk17? as long as we use your 17-based-images
[18:41:13]	gbrown	yes, that's right

Comment 13

[Armen [:armenzg]]

That's helpful. Thanks!