894507 - Document Android x86 emulator setup

Status: RESOLVED WORKSFORME

(Testing :: General, defect)

Description

[Geoff Brown [:gbrown]]

dminor and I have worked through a few problems setting up the Android x86 emulator for unit tests on ix. We have notes but should organize them and make sure they are complete. I'll do that here, and likely transfer to a wiki later.

Comment 1

[Geoff Brown [:gbrown]]

The goal here is to run multiple (probably 4 to 6) Android x86 emulators on each test slave, configured appropriately to run Firefox for Android and associated unit tests.

The overall strategy is to create one emulator avd manually with all the necessary settings, make multiple copies of that avd, copy the group of avd definitions to each test slave, and then run emulators against each avd.

1. Create one master AVD.

This step need not be run on a test slave. It can be run on any computer that has the Android SDK installed. 

 - Make sure the Android SDK is installed.
 - Run "android" from <android sdk>/tools/android -- this brings up a UI for the "Android SDK Manager".
 - In the Android SDK Manager, verify that an option for "Android 4.2.2 (API 17)" is available and has installed components:
     -- "SDK Platform"
     -- "Intel x86 Atom System Image"
 - In the Android SDK Manager, select "Tools" > "Manage AVDs" to launch the "Android Virtual Device Manager"
 - In the Android Virtual Device Manager, on the "Android Virtual Devices" tab, select "New..." and enter:
    -- AVD Name: test-x86-1
    -- Device: Galaxy Nexus
    -- Target: Android 4.2.2 - API Level 17
    -- CPU: Intel Atom (x86)
    -- RAM: 1024
    -- VMHeap: 64
    -- Internal Storage: 600
    -- SDCard: 500
    -- Use Host GPU: checked
 - Select OK twice to accept the settings and create the AVD.
 - Exit the Android Virtual Device Manager.
 - Exit the Android SDK Manager.

2. Customize the master AVD.

Execute this procedure on the same computer as step 1.

Here we pre-install SUTagent, Watcher, FenCP, and FfxCP and customized su.

 - obtain an appropriate x86 build of su, either from orangutan (https://github.com/wlach/orangutan), or an existing build (I'll attach one to this bug)
 - download an appropriate tests zip, such as:  ftp://ftp.mozilla.org/pub/mobile/nightly/latest-mozilla-central-android-x86/fennec-25.0a1.multi.android-i386.tests.zip
 - extract the bin directory from the zip to obtain the necessary apks
 - launch the emulator with the new AVD, created in step 1:
    -- emulator -avd test-x86-1 -partition-size 1024
 - install the apks:
    -- adb -e install Watcher.apk
    -- adb -e install sutAgentAndroid.apk
    -- adb -e install FenCP.apk
    -- adb -e install FfxCP.apk
 - over-write /system/xbin/su:
    -- adb -e shell mount -o remount,rw /system
    -- adb -e push su /system/xbin
    -- adb -e shell ls -l /system/xbin/su
    -- adb -e shell chmod 6755 /system/xbin/su
    -- adb -e shell mount -o remount,ro /system
 - with the emulator still running, find the system image on the computer, typically /tmp/android-{user}/emulator-{xxxxxx} and make a temporary copy of it
    -- cp /tmp/android-mozdev/emulator* /tmp/system.img
 - close the emulator
 - move the temporary copy of the system image to the avd:
    -- mv /tmp/system.img ~/.android/avd/test-x86-1.avd
3. Make copies of the master AVD.
 
Execute this procedure on the same computer as step 1.

Copy test-x86-1.avd to a new directory to create test-x86-2.avd, etc. Paths need to be corrected for:
 - change in avd path between step 1 computer and test slave
 - change in paths between each avd

A simple script takes care of this:
  -- ./copyavds.sh

4. Transfer the avd definitions to the test slave.

On the same computer as in step 1:
   -- cd ~/.android/avd
   -- tar cvfz test-avds.tgz test-x86*
   -- <transfer to test slave>

On the test slave:
   -- cd ~/.android/avd
   -- tar xvfz test-avds.tgz

5. Launch the emulators on the test slave.

Basically we launch "emulator -avd <avd> -port <port", then use tcp redirection to manage ports. There's a python script for this:
   -- ./launch.py

Comment 2

[Geoff Brown [:gbrown]]

I missed (at least!) one item:

At the end of step 2, restart the emulator to update the avd definition with the system image path and size. Close it again.

Comment 3

[Geoff Brown [:gbrown]]

Attachment: archive containing su suitable for step 2

Comment 4

[Geoff Brown [:gbrown]]

Attachment: script I used for step 3

Comment 5

[Geoff Brown [:gbrown]]

The installation of FenCP and FfxCP in step 2 is not necessary and can be problematic: it can cause Fennec installation failures due to certificate mismatches. These are not currently installed on the pandas -- let's not install them for x86 either.

Comment 6

[Geoff Brown [:gbrown]]

Here is a replacement for Comment 1, with corrections from Comment 2 and Comment 5.


The goal is to run 4 Android x86 emulators on each test slave, configured appropriately to run Firefox for Android and associated unit tests.

The overall strategy is to create one emulator avd manually with all the necessary settings, make multiple copies of that avd, copy the group of avd definitions to each test slave, and then run emulators against each avd.

1. Create one master AVD.

This step need not be run on a test slave. It can be run on any computer that has the Android SDK installed. 

 - Install the Android SDK if not already available.
 - Run "android" from <android sdk>/tools/android -- this brings up a UI for the "Android SDK Manager".
 - In the Android SDK Manager, verify that an option for "Android 4.2.2 (API 17)" is available and has installed components:
     -- "SDK Platform"
     -- "Intel x86 Atom System Image"
 - In the Android SDK Manager, select "Tools" > "Manage AVDs" to launch the "Android Virtual Device Manager"
 - In the Android Virtual Device Manager, on the "Android Virtual Devices" tab, select "New..." and enter:
    -- AVD Name: test-x86-1
    -- Device: Galaxy Nexus
    -- Target: Android 4.2.2 - API Level 17
    -- CPU: Intel Atom (x86)
    -- RAM: 1024
    -- VMHeap: 64
    -- Internal Storage: 600
    -- SDCard: 500
    -- Use Host GPU: checked
 - Select OK twice to accept the settings and create the AVD.
 - Exit the Android Virtual Device Manager.
 - Exit the Android SDK Manager.

2. Customize the master AVD.

Execute this procedure on the same computer as step 1.

Here we pre-install SUTagent and Watcher and customized su.

 - obtain an appropriate x86 build of su, either from orangutan (https://github.com/wlach/orangutan), or an existing build (there is one attached to this bug)
 - download an appropriate tests zip, such as:  ftp://ftp.mozilla.org/pub/mobile/nightly/latest-mozilla-central-android-x86/fennec-25.0a1.multi.android-i386.tests.zip
 - extract the bin directory from the zip to obtain the necessary apks
 - launch the emulator with the new AVD, created in step 1:
    -- emulator -avd test-x86-1 -partition-size 1024
 - install the apks:
    -- adb -e install Watcher.apk
    -- adb -e install sutAgentAndroid.apk
 - verify correct installation by starting Watcher and SUTAgent manually (click on the application icons)
 - over-write /system/xbin/su:
    -- adb -e shell mount -o remount,rw /system
    -- adb -e push su /system/xbin
    -- adb -e shell ls -l /system/xbin/su
    -- adb -e shell chmod 6755 /system/xbin/su
    -- adb -e shell mount -o remount,ro /system
 - with the emulator still running, find the system image on the computer, typically /tmp/android-{user}/emulator-{xxxxxx} and make a temporary copy of it
    -- cp /tmp/android-mozdev/emulator* /tmp/system.img
 - close the emulator
 - move the temporary copy of the system image to the avd:
    -- mv /tmp/system.img ~/.android/avd/test-x86-1.avd
 - launch the emulator again to update the path and size of the system.img:
    -- emulator -avd test-x86-1
 - close the emulator

3. Make copies of the master AVD.
 
Execute this procedure on the same computer as step 1.

Copy test-x86-1.avd to a new directory to create test-x86-2.avd, etc. Paths need to be corrected for:
 - change in avd path between step 1 computer and test slave
 - change in paths between each avd

A simple script takes care of this (may need customization for your environment):
  -- ./copyavds.sh

4. Transfer the avd definitions to the test slave.

On the same computer as in step 1:
   -- cd ~/.android/avd
   -- tar cvfz test-avds.tgz test-x86*
   -- <transfer to test slave>

On the test slave:
   -- cd ~/.android/avd
   -- tar xvfz test-avds.tgz

5. Launch the emulators on the test slave.

We launch "emulator -avd <avd> -port <port", then use tcp redirection to manage ports. There's a python script for this:
   -- ./launch.py

After a few minutes, launch.py will start 4 emulator instances. You can verify operation with something like:

$ adb devices
List of devices attached 
emulator-5554	device
emulator-5556	device
emulator-5558	device
emulator-5560	device

$ telnet 127.0.0.1 20701
Trying 127.0.0.1...
Connected to 127.0.0.1.
Escape character is '^]'.
$>ps
10037	1595	com.android.exchange
10047	1697	com.mozilla.SUTAgentAndroid
10038	1623	com.android.providers.calendar
10027	1660	com.android.calendar
10033	1270	com.android.systemui
10005	1431	com.android.location.fused
10018	1360	com.android.inputmethod.latin
1000	1204	system
10002	1396	com.android.launcher
10046	1644	com.mozilla.watcher
1001	1381	com.android.phone

...for each emulator (ports 20701, 20702, 20703, 20704).

Comment 7

[Geoff Brown [:gbrown]]

Attachment: launch.py

Python script to launch 4 emulators and redirect sut ports appropriately.

Comment 8

[Geoff Brown [:gbrown]]

Attachment: launch.py

Updated script redirects both sut ports.

test-x86-1: 5554; sut port:20701/20700
test-x86-2: 5556; sut port:20703/20702
test-x86-3: 5558; sut port:20705/20704
test-x86-4: 5560; sut port:20707/20706

Comment 9

[Geoff Brown [:gbrown]]

Many reftests fail unless the device's screen characteristics are similar to a tegra: https://bugzilla.mozilla.org/show_bug.cgi?id=892688#c1. Using a tegra-like screen in the emulator seems to help some mochitests also. This makes the emulator setup a little more complicated, so here are all the instructions again, with a few additions. 

These instructions should be used in preference to those in comment 1 and comment 6.


The goal is to run 4 Android x86 emulators on each test slave, configured appropriately to run Firefox for Android and associated unit tests.

The overall strategy is to create one emulator avd manually with all the necessary settings, make multiple copies of that avd, copy the group of avd definitions to each test slave, and then run emulators against each avd.

1. Create one master AVD.

This step need not be run on a test slave. It can be run on any computer that has the Android SDK installed. 

 - Install the Android SDK if not already available. See http://developer.android.com/sdk/index.html
 - Run "android" from <android sdk>/tools/android -- this brings up a UI for the "Android SDK Manager".
 - In the Android SDK Manager, verify that an option for "Android 4.2.2 (API 17)" is available and has installed components:
     -- "SDK Platform"
     -- "Intel x86 Atom System Image"
 - In the Android SDK Manager, select "Tools" > "Manage AVDs" to launch the "Android Virtual Device Manager"
 - In the Android Virtual Device Manager, on the "Device Definitions" tab, select "New Device..." and on the "Create New Device" dialog enter:
    -- Name: mozilla-device
    -- Input: Keyboard -- checked
    -- RAM: 1024 MiB
    -- Resolution: 1024 x 768
    -- Screen size (in): 7
 - Save this device definition.
 - In the Android Virtual Device Manager, on the "Android Virtual Devices" tab, select "New..." and enter:
    -- AVD Name: test-x86-1
    -- Device: mozilla-device
    -- Target: Android 4.2.2 - API Level 17
    -- CPU: Intel Atom (x86)
    -- RAM: 1024
    -- VMHeap: 64
    -- Internal Storage: 600
    -- SDCard: 500
    -- Use Host GPU: checked
 - Select OK twice to accept the settings and create the AVD.
 - Exit the Android Virtual Device Manager.
 - Exit the Android SDK Manager.

2. Customize the master AVD.

Execute this procedure on the same computer as step 1.

Here we pre-install SUTagent and Watcher and customized su.

 - obtain an appropriate x86 build of su, either from orangutan (https://github.com/wlach/orangutan), or an existing build (there is one attached to this bug)
 - download an appropriate tests zip, such as:  ftp://ftp.mozilla.org/pub/mobile/nightly/latest-mozilla-central-android-x86/fennec-25.0a1.multi.android-i386.tests.zip
 - extract the bin directory from the zip to obtain the necessary apks
 - launch the emulator with the new AVD, created in step 1:
    -- emulator -avd test-x86-1 -partition-size 1024
 - install the apks:
    -- adb -e install Watcher.apk
    -- adb -e install sutAgentAndroid.apk
 - verify correct installation by starting Watcher and SUTAgent manually (click on the application icons)
 - over-write /system/xbin/su:
    -- adb -e shell mount -o remount,rw /system
    -- adb -e push su /system/xbin
    -- adb -e shell chmod 6755 /system/xbin/su
    -- adb -e shell ls -l /system/xbin/su
    -- adb -e shell mount -o remount,ro /system
 - with the emulator still running, find the system image on the computer, typically /tmp/android-{user}/emulator-{xxxxxx} and make a temporary copy of it
    -- cp /tmp/android-mozdev/emulator* /tmp/system.img
 - close the emulator
 - move the temporary copy of the system image to the avd:
    -- mv /tmp/system.img ~/.android/avd/test-x86-1.avd
 - launch the emulator again to update the path and size of the system.img:
    -- emulator -avd test-x86-1
 - close the emulator

3. Make copies of the master AVD.
 
Execute this procedure on the same computer as step 1.

Copy test-x86-1.avd to a new directory to create test-x86-2.avd, etc. Paths need to be corrected for:
 - change in avd path between step 1 computer and test slave
 - change in paths between each avd

A simple script takes care of this (may need customization for your environment):
  -- ./copyavds.sh

4. Transfer the avd definitions to the test slave.

On the same computer as in step 1:
   -- cd ~/.android/avd
   -- tar cvfz test-avds.tgz test-x86*
   -- <transfer to test slave>

On the test slave:
   -- cd ~/.android/avd
   -- tar xvfz test-avds.tgz

5. Prepare the test slave.

The Android emulator is required on the test slave. To verify that the emulator is running, adb is useful. Also the test scripts in use today use adb (minimally). The easiest way to get these tools on the test slave is to download and install the Android SDK from http://developer.android.com/sdk/index.html. Then update your environment:

 -- set PATH=$PATH:<android-sdk>/tools:<android-sdk>/platform-tools
 -- set DISPLAY=:0.0

6. Launch the emulators on the test slave.

We launch "emulator -avd <avd> -port <port", then use tcp redirection to manage ports. There's a python script for this (attached):
   -- ./launch.py

After a few minutes, launch.py will start 4 emulator instances and print the names and ports associated with each:

test-x86-1: 5554; sut port:20701/20700
test-x86-2: 5556; sut port:20703/20702
test-x86-3: 5558; sut port:20705/20704
test-x86-4: 5560; sut port:20707/20706

You can verify operation with something like:

$ adb devices
List of devices attached 
emulator-5554	device
emulator-5556	device
emulator-5558	device
emulator-5560	device

$ telnet 127.0.0.1 20701
Trying 127.0.0.1...
Connected to 127.0.0.1.
Escape character is '^]'.
$>ps
10037	1595	com.android.exchange
10047	1697	com.mozilla.SUTAgentAndroid
10038	1623	com.android.providers.calendar
10027	1660	com.android.calendar
10033	1270	com.android.systemui
10005	1431	com.android.location.fused
10018	1360	com.android.inputmethod.latin
1000	1204	system
10002	1396	com.android.launcher
10046	1644	com.mozilla.watcher
1001	1381	com.android.phone

...for each emulator (ports 20701, 20703, 20705, 20707).

Comment 10

[Armen [:armenzg]]

(In reply to Geoff Brown [:gbrown] from comment #9)
> 5. Prepare the test slave.
> 
> The Android emulator is required on the test slave. To verify that the
> emulator is running, adb is useful. Also the test scripts in use today use
> adb (minimally). The easiest way to get these tools on the test slave is to
> download and install the Android SDK from
> http://developer.android.com/sdk/index.html. Then update your environment:
> 
>  -- set PATH=$PATH:<android-sdk>/tools:<android-sdk>/platform-tools
>  -- set DISPLAY=:0.0
> 

I think the best way forward is to add mock support to the androix86 mozharness script so we can add the android sdk. We don't want to have to be supporting the installation of the SDKs through puppet.

> 6. Launch the emulators on the test slave.
> 
> We launch "emulator -avd <avd> -port <port", then use tcp redirection to
> manage ports. There's a python script for this (attached):
>    -- ./launch.py
> 
> After a few minutes, launch.py will start 4 emulator instances and print the
> names and ports associated with each:
> 
> test-x86-1: 5554; sut port:20701/20700
> test-x86-2: 5556; sut port:20703/20702
> test-x86-3: 5558; sut port:20705/20704
> test-x86-4: 5560; sut port:20707/20706
> 
> You can verify operation with something like:
> 
> $ adb devices
> List of devices attached 
> emulator-5554	device
> emulator-5556	device
> emulator-5558	device
> emulator-5560	device
> 
> $ telnet 127.0.0.1 20701
> Trying 127.0.0.1...
> Connected to 127.0.0.1.
> Escape character is '^]'.
> $>ps
> 10037	1595	com.android.exchange
> 10047	1697	com.mozilla.SUTAgentAndroid
> 10038	1623	com.android.providers.calendar
> 10027	1660	com.android.calendar
> 10033	1270	com.android.systemui
> 10005	1431	com.android.location.fused
> 10018	1360	com.android.inputmethod.latin
> 1000	1204	system
> 10002	1396	com.android.launcher
> 10046	1644	com.mozilla.watcher
> 1001	1381	com.android.phone
> 
> ...for each emulator (ports 20701, 20703, 20705, 20707).

We'll have to port launch.py to mozharness.

Are we 100% that we want to be going first with 4 emulator jobs?
I would prefer to make it work with 1, go live on Cedar and then focus on adding 4.
Running 4 jobs on the same machine is not something that we have done before and we could find new edge cases with buildbot/mozharness.

Comment 11

[Geoff Brown [:gbrown]]

I am confident that 4 emulators can run at the same time, but I agree that we may encounter intermittent failures with 4 emulators + buildbot/mozharness. 1 at a time on Cedar to start seems reasonable...we only need the 4 for capacity in continuous integration.

Comment 12

[Geoff Brown [:gbrown]]

An additional modification is necessary to allow for xpcshell tests: In Comment 9, step 2, after "- over-write /system/xbin/su:", 

 - adb -e shell chmod 777 /data/local

Comment 13

[Geoff Brown [:gbrown]]

Attachment: launch.py

Updated launch.py to specify kernel, system, and ramdisk image files from new avd tarball: see https://bugzilla.mozilla.org/show_bug.cgi?id=895186#c29.

Comment 14

[Geoff Brown [:gbrown]]

Disregard comment 12 -- now addressed by bug 902645.

Comment 15

[Armen [:armenzg]]

gbrown, would you say that the documentation is up-to-date?
Does in contain our last conversations wrt to android-17 vs android-18?

I'm doing a swipe at the bug under the tracking bug to see what is left besides bug 895186.

Comment 16

[Geoff Brown [:gbrown]]

Comment 9 is still valid and current, and correctly references android-17 system images regardless of the version of Android SDK Manager in use. Let's add this for clarity:

On the test machines, the Android emulator must be installed. The emulator is part of the Android SDK, so the easiest way to get the emulator is to install the Android SDK. Typically, only the most recent version of the Android SDK is available for download -- older versions are not generally archived. However, the Android SDK Manager can be used to selectively install components from earlier versions.  We need to use the android-17 or newer version of the emulator. The emulator tends to improve over time, so the latest emulator is best. At this time, we have tested with both the android-17 and android-18 emulators and have noticed little difference between them; the android-18 emulator is recommended.

Comment 17

[Graydon Hoare :graydon]

(In reply to Geoff Brown [:gbrown] from comment #16)
> Comment 9 is still valid and current

Hi,

Reading through these steps (and replicating them to build images again, as I tried last week in bug 910092) I notice two differences between them and the assumptions in the mozharness scripts and configs associated with emulator-based testing. Wondering if you could clarify here:

2. Comment 9 suggests copying system.img to ~/.android/avd/test-x86-1.avd, whereas in the mozharness androidx86.py config, we have this line https://hg.mozilla.org/build/mozharness/file/00028945f6b1/configs/android/androidx86.py#l74 which specifies a shared system.img between the AVDs. I thought this might be a typo (as did Armen) but later in Comment 9 there's the step "launch the emulator again to update the path and size of the system.img", suggesting the emulators are directly aware of their (separate, per-AVD) system.img files (as indeed they are, if I check the associated test-foo.avd/hardware-qemu.ini files). It seems to me we should settle on one arrangement or the other here; which should it be?

2. Comment 9 does not mention bundling up qemu-kernel or ramdisk.img, despite the mozharness androida86.py script specifying them and passing them to the emulator it runs. They are present in the AVD tarball we've been testing with as well as the one I made, but I wonder if they're necessary / important.

Appreciate any suggestions, thanks.

Comment 18

[Geoff Brown [:gbrown]]

Around Comment 13 we decided to use shared images, making Comment 9 slightly inaccurate.

It is probably still best to initially copy system.img to ~/.android/avd/test-x86-1.avd. When the emulator is subsequently launched, it will use that automatically without any changes to the ini files. However, since system.img is large and read-only, we can save space by using the same system.img from all of the emulators. So in step 3, I think system.img should be moved up a level (~/.android/avd), and the ini file modified to point to the new location.

There is less need for having qemu-kernel and ramdisk.img in the tarball, if they are available elsewhere (installed with the SDK) but we thought it best not to rely on a particular SDK image installation. So we need to copy thoses images from the SDK (we used the android-17 images) to ~/.android/avd before creating the tar. 

I will run through the procedure again and write up a new version of the instructions and attachments.

Comment 19

[Geoff Brown [:gbrown]]

These instructions should be used in preference to those in comment 1, comment 6, and comment 9. (This revision discusses the shared system images and also changes the device resolution to 1024x816, so that the resolution reported by sutagent and tests is 1024x768 -- same as a tegra.) 


The goal is to run 4 Android x86 emulators on each test slave, configured appropriately to run Firefox for Android and associated unit tests.

The overall strategy is to create one emulator avd manually with all the necessary settings, make multiple copies of that avd, copy the group of avd definitions to each test slave, and then run emulators against each avd.

1. Create one master AVD.

This step need not be run on a test slave. It can be run on any computer that has the Android SDK installed. 

 - Install the Android SDK if not already available. See http://developer.android.com/sdk/index.html
 - Run "android" from <android sdk>/tools/android -- this brings up a UI for the "Android SDK Manager".
 - In the Android SDK Manager, verify that an option for "Android 4.2.2 (API 17)" is available and has installed components:
     -- "SDK Platform"
     -- "Intel x86 Atom System Image"
 - In the Android SDK Manager, select "Tools" > "Manage AVDs" to launch the "Android Virtual Device Manager"
 - In the Android Virtual Device Manager, on the "Device Definitions" tab, select "New Device..." and on the "Create New Device" dialog enter:
    -- Name: mozilla-device
    -- Input: Keyboard -- checked
    -- RAM: 1024 MiB
    -- Resolution: 1024 x 816
    -- Screen size (in): 7
    -- Density: mdpi
 - Save this device definition.
 - In the Android Virtual Device Manager, on the "Android Virtual Devices" tab, select "New..." and enter:
    -- AVD Name: test-x86-1
    -- Device: mozilla-device
    -- Target: Android 4.2.2 - API Level 17
    -- CPU: Intel Atom (x86)
    -- RAM: 1024
    -- VMHeap: 64
    -- Internal Storage: 600
    -- SDCard: 500
    -- Use Host GPU: checked
 - Select OK twice to accept the settings and create the AVD. (This may take a minute.)
 - Exit the Android Virtual Device Manager.
 - Exit the Android SDK Manager.

2. Customize the master AVD.

Execute this procedure on the same computer as step 1.

Here we pre-install SUTagent and Watcher and customized su.

 - obtain an appropriate x86 build of su, either from orangutan (https://github.com/wlach/orangutan), or an existing build (there is one attached to this bug)
 - download an appropriate tests zip, such as:  ftp://ftp.mozilla.org/pub/mobile/nightly/latest-mozilla-central-android-x86/en-US/fennec-27.0a1.en-US.android-i386.tests.zip
 - extract the bin directory from the tests zip to obtain the necessary apks
 - launch the emulator with the new AVD, created in step 1 (either ensure that the android sdk emulator is in your PATH, or specify the path; for some platforms, you may need to specify an emulator variant, like emulator64-x86):
    -- emulator -avd test-x86-1 -partition-size 1024
 - wait for the emulator to start (it may take a few minutes)
 - when the Android welcome screen is displayed, click OK
 - over-write /system/xbin/su:
    -- adb -e shell mount -o remount,rw /system
    -- adb -e push su /system/xbin
    -- adb -e shell chmod 6755 /system/xbin/su
    -- adb -e shell ls -l /system/xbin/su
       --- expect: -rwsr-sr-x root     root        17748 2013-09-19 17:48 su
    -- adb -e shell mount -o remount,ro /system
 - install the apks:
    -- adb -e install Watcher.apk
    -- adb -e install sutAgentAndroid.apk
 - verify correct installation by starting Watcher and SUTAgent manually (click on the application icons)
 - with the emulator still running, find the system image on the computer, typically /tmp/android-{user}/emulator-{xxxxxx} and make a temporary copy of it
    -- cp /tmp/android-mozdev/emulator* /tmp/system.img
 - close the emulator
 - move the temporary copy of the system image to the avd:
    -- mv /tmp/system.img ~/.android/avd/test-x86-1.avd
 - launch the emulator again to update the path and size of the system.img in the avd ini files:
    -- emulator -avd test-x86-1
 - wait for the emulator to start, then close the emulator

3. Make copies of the master AVD.
 
Execute this procedure on the same computer as step 1.

Copy test-x86-1.avd to a new directory to create test-x86-2.avd, etc. Paths need to be corrected for:
 - change in avd path between step 1 computer and test slave
 - change in paths between each avd

A simple script takes care of this (needs customization for your environment):
  -- ./copyavds.sh

After executing, we want 4 avd definitions. Each hardware-qemu has pointers to img files in the avd directory and pointers to img files in the avd root directory:

  hw.sdCard.path = /home/cltbld/.android/avd/test-x86-1.avd/sdcard.img
  disk.cachePartition.path = /home/cltbld/.android/avd/test-x86-1.avd/cache.img
  kernel.path = /home/cltbld/.android/avd/kernel-qemu
  disk.ramdisk.path = /home/cltbld/.android/avd/ramdisk.img
  disk.systemPartition.initPath = /home/cltbld/.android/avd/system.img
  disk.dataPartition.path = /home/cltbld/.android/avd/test-x86-1.avd/userdata-qemu.img

Note that kernel-qemu, ramdisk.img and system.img are in the .android/avd directory, while copies of sdcard.img, cache.img, and userdata-qemu.img are in each test avd directory.

4. Transfer the avd definitions to the test slave.

On the same computer as in step 1:
   -- cd ~/.android/avd
   -- tar cvfz test-avds.tar.gz test-x86* kernel-qemu ramdisk.img system.img
   -- <transfer to test slave>

On the test slave:
   -- cd ~/.android/avd
   -- tar xvfz test-avds.tar.gz

5. Prepare the test slave.

The Android emulator is required on the test slave. To verify that the emulator is running, adb is useful. Also the test scripts in use today use adb. The easiest way to get these tools on the test slave is to download and install the Android SDK from http://developer.android.com/sdk/index.html. Then update your environment:

 -- set PATH=$PATH:<android-sdk>/tools:<android-sdk>/platform-tools
 -- set DISPLAY=:0.0

6. Launch the emulators on the test slave.

A mozharness script normally launches the emulator instances. See https://hg.mozilla.org/build/mozharness/file/d080fb4225a3/scripts/androidx86_emulator_unittest.py.

To launch the emulator manually, run 

   -- emulator -avd <avd> -port <port> -debug all -kernel <kernel_path>
         -system <system_image_path> -ramdisk <ramdisk_path>

Then use tcp redirection to manage ports for sutagent. 

There's a python script for this (attached):
   -- ./launch.py

We found that launching multiple emulators at once was unreliable (some instances would fail intermittently, without reporting an error), so this script sleeps between each process launch.

After a few minutes, launch.py will start 4 emulator instances and print the names and ports associated with each:

test-x86-1: 5554; sut port:20701/20700
test-x86-2: 5556; sut port:20703/20702
test-x86-3: 5558; sut port:20705/20704
test-x86-4: 5560; sut port:20707/20706

You can verify operation with something like:

$ adb devices
List of devices attached 
emulator-5554	device
emulator-5556	device
emulator-5558	device
emulator-5560	device

$ telnet 127.0.0.1 20701
Trying 127.0.0.1...
Connected to 127.0.0.1.
Escape character is '^]'.
$>ps
10037	1595	com.android.exchange
10047 	1697	com.mozilla.SUTAgentAndroid
10038	1623	com.android.providers.calendar
10027	1660	com.android.calendar
10033	1270	com.android.systemui
10005	1431	com.android.location.fused
10018	1360	com.android.inputmethod.latin
1000	1204	system
10002	1396	com.android.launcher
10046	1644	com.mozilla.watcher
1001	1381	com.android.phone

...for each emulator (ports 20701, 20703, 20705, 20707).

Comment 20

[Geoff Brown [:gbrown]]

Attachment: script for step 3

Comment 21

[Geoff Brown [:gbrown]]

Attachment: launch.py

Comment 22

[Geoff Brown [:gbrown]]

I ran through the entire procedure again, created new avds and tested them: http://people.mozilla.org/~gbrown/test-avds-sep19.tar.gz. (I filed bug 919812 to have these new avds distributed.)

Comment 19 is the final word on avd creation (hopefully!).