1438351 - Experiment with running android mochi/reftests on packet.net

Status: RESOLVED FIXED

(Testing :: General, defect, P1)

Description

[Geoff Brown [:gbrown]]

I am experimenting with the android emulator on packet.net. I'll use this bug to capture some notes.

Comment 1

[Geoff Brown [:gbrown]]

After signing on to packet.net with Joel's credentials (mine were delayed), I deployed a type 0 server running Ubuntu 16.04 following https://help.packet.net/quick-start/deploy-a-server

I tried "kvm-ok" first -- command not found.

apt-get install qemu-kvm libvirt-bin ubuntu-vm-builder bridge-utils -- failed
apt-get update -- okay
apt-get install qemu-kvm libvirt-bin ubuntu-vm-builder bridge-utils -- okay

Now:

root@gbrownubuntu1604t0:~# kvm-ok
INFO: /dev/kvm exists
KVM acceleration can be used

Great!

I downloaded our emulator and avd from tooltool:

wget https://raw.githubusercontent.com/mozilla/build-tooltool/master/tooltool.py
python tooltool.py fetch --manifest emulator.manifest --authentication-file ttauth
python tooltool.py fetch --manifest avd.manifest

export PATH=~/android-sdk-linux/emulator:~/android-sdk-linux/tools:~/android-sdk-linux/platform-tools:$PATH

and edited test-1.ini paths.

Emulator wouldn't start -- missing libGL.so -- so

apt-get install libgl1-mesa-glx

Now:

root@gbrownubuntu1604t0:~# emulator -list-avds
test-1
root@gbrownubuntu1604t0:~# emulator -accel-check
accel:
0
KVM (version 12) is installed and usable.
accel

Great!!

But:

root@gbrownubuntu1604t0:~# emulator -avd test-1 -skip-adb-auth -verbose -show-kernel

segfaulted, complaining about the gpu.


root@gbrownubuntu1604t0:~# emulator -avd test-1 -skip-adb-auth -verbose -show-kernel -gpu swiftshader

got much further but still did not start, complaining

QXcbConnection: Could not connect to display 


apt-get install xvfb
xvfb-run emulator ... > emulator.log &

root@gbrownubuntu1604t0:~# adb devices
List of devices attached
emulator-5554    device

root@gbrownubuntu1604t0:~# adb shell
generic_x86_64:/ $

Comment 2

[Geoff Brown [:gbrown]]

apt-get install python-pip
pip install virtualenv
virtualenv venv
cd venv/bin
. activate
pip install mozterm
pip install six

Comment 3

[Geoff Brown [:gbrown]]

python tooltool.py fetch --manifest hostutils.manifest
mkdir hostutils
cd hostutils
tar xvf ../host-utils-60.0a2.en-US.linux-x86_64.tar.gz

Comment 4

[Geoff Brown [:gbrown]]

My manifests:

(venv) root@gbrownubuntu1604t0:~# cat hostutils.manifest
[
  {
    "size": 74983178,
    "visibility": "public",
    "digest": "800acc77951dbd47880ded6ebfb70b2b6a1e034a52cb0e84fda1931e1825152853c0b5cfa4d05afdc2038f30c1f077fd037d9a49751505d98cbeb2b7e0377328",
    "algorithm": "sha512",
    "filename": "host-utils-60.0a2.en-US.linux-x86_64.tar.gz",
    "unpack": true
  }
]
(venv) root@gbrownubuntu1604t0:~# cat avd.manifest
[
  {
    "size": 445624230,
    "visibility": "public",
    "digest": "fb951185e21ba6124d8c8ba42202613312401c65e56d52480cc08c87816936b9dd00024f4fc00183ee0e199c91ed5b9209867983c1d13daf5b34bcdc14ee9fb6",
    "algorithm": "sha512",
    "filename": "AVDs-x86-android-7.0-build-2017-12-08.tar.gz",
    "unpack": true
  }
]

(venv) root@gbrownubuntu1604t0:~# cat emulator.manifest
        [
        {
        "size": 237849484,
        "visibility": "internal",
        "digest": "596db3063758aea49f441ed3dccc5429408e037ba953ab8ea676969c12ed267416a7b1f7d61ad5a06fd2c0fe234131843bd9c57a33b8dd8677aa0b25bbb799d7",
        "algorithm": "sha512",
        "filename": "android-sdk_r26b-linux.tar.gz",
        "unpack": "True"
        }
        ]

Note that emulator has internal visibility -- use an appropriate auth token.

Comment 5

[Geoff Brown [:gbrown]]

Downloaded Android x86 target.apk and test zips from mozilla-central.

adb install -r target.apk
mkdir tests
unzip the test zips in tests
first run of runtestsremote.py failed: xpcshell crashed, missing libgtk.so

apt-get install libgtk-3-0

python -u ~/tests/mochitest/runtestsremote.py --app=org.mozilla.fennec_aurora --remote-webserver=10.0.2.2 --xre-path=~/hostutils/host-utils-60.0a2.en-US.linux-x86_64 --utility-path=~/hostutils/host-utils-60.0a2.en-US.linux-x86_64 --certificate-path=~/tests/certs --symbols-path=https://queue.taskcluster.net/v1/task/Gb94v26tTWqfrr18Jni3Tg/artifacts/public/build/target.crashreporter-symbols.zip --quiet --extra-profile-file=fonts --extra-profile-file=hyphenation --screenshot-on-fail --flavor=chrome
...
TEST-SKIP | xpfe/appshell/test/test_windowlessBrowser.xul | took 0ms
Running manifest: devtools/shared/security/tests/chrome/chrome.ini
pushing /root/tests/mochitest/ to /mnt/sdcard/tests/chrome on device...
runtests.py | Failed to copy /root/fonts to profile
runtests.py | Failed to copy /root/hyphenation to profile
pk12util: PKCS12 IMPORT SUCCESSFUL
MochitestServer : launching [u'/root/hostutils/host-utils-60.0a2.en-US.linux-x86_64/xpcshell', '-g', '/root/hostutils/host-utils-60.0a2.en-US.linux-x86_64', '-f', '/root/hostutils/host-utils-60.0a2.en-US.linux-x86_64/components/httpd.js', '-e', "const _PROFILE_PATH = '/tmp/tmpE9FWm2.mozrunner'; const _SERVER_PORT = '8888'; const _SERVER_ADDR = '10.0.2.2'; const _TEST_PREFIX = undefined; const _DISPLAY_RESULTS = false;", '-f', '/root/tests/mochitest/server.js']
runtests.py | Server pid: 16965
runtests.py | Websocket server pid: 16968
runtests.py | SSL tunnel pid: 16971
runtests.py | Running with e10s: False
runtests.py | Running tests: start.

INFO | automation.py | Application pid: 2310
INFO | automation.py | Application ran for: 0:00:17.730436
INFO | zombiecheck | Reading PID log: /tmp/tmpbavRgbpidlog
MOZ_UPLOAD_DIR not defined; tombstone check skipped
Automation Error: No crash directory (/mnt/sdcard/tests/profile/minidumps) found on remote device
Stopping web server
Stopping web socket server
Stopping ssltunnel
leakcheck | refcount logging is off, so leaks can't be detected!
runtests.py | Running tests: end.
Unable to retrieve log file (/mnt/sdcard/tests/logs/mochitest.log) from remote device
Buffered messages finished
Running manifest: dom/animation/test/chrome.ini
runtests.py | Failed to copy /root/fonts to profile
runtests.py | Failed to copy /root/hyphenation to profile
pk12util: PKCS12 IMPORT SUCCESSFUL
MochitestServer : launching [u'/root/hostutils/host-utils-60.0a2.en-US.linux-x86_64/xpcshell', '-g', '/root/hostutils/host-utils-60.0a2.en-US.linux-x86_64', '-f', '/root/hostutils/host-utils-60.0a2.en-US.linux-x86_64/components/httpd.js', '-e', "const _PROFILE_PATH = '/tmp/tmpaZaP4u.mozrunner'; const _SERVER_PORT = '8888'; const _SERVER_ADDR = '10.0.2.2'; const _TEST_PREFIX = undefined; const _DISPLAY_RESULTS = false;", '-f', '/root/tests/mochitest/server.js']
runtests.py | Server pid: 17109
runtests.py | Websocket server pid: 17112
runtests.py | SSL tunnel pid: 17115
runtests.py | Running with e10s: False
runtests.py | Running tests: start.

INFO | automation.py | Application pid: 0
INFO | automation.py | Application ran for: 0:00:14.435885
INFO | zombiecheck | Reading PID log: /tmp/tmpI9qtq_pidlog
MOZ_UPLOAD_DIR not defined; tombstone check skipped
PROCESS-CRASH | remoteautomation.py | java-exception Could not open database at android.database.sqlite.SQLiteConnection.nativeOpen(Native Method)
Stopping web server
Stopping web socket server
Stopping ssltunnel
leakcheck | refcount logging is off, so leaks can't be detected!
runtests.py | Running tests: end.

Comment 6

[Geoff Brown [:gbrown]]

(In reply to Geoff Brown [:gbrown] from comment #5)

I got stuck here. Firefox was crashing on startup and my fanciful imagination led me to investigate various unlikely scenarios before I remembered: runtime permissions!

> Downloaded Android x86 target.apk and test zips from mozilla-central.
> 
> adb install -r target.apk

adb install -r -g target.apk !!

Comment 7

[Geoff Brown [:gbrown]]

Now I can start an Android 7.0 x86 emulator on a type 0 or type 1 instance and run various tests, semi-manually (I use custom scripts to run the emulator and run the test harnesses, based on manual download/install of a build + test zips:  I'm not using docker and I'm not using mozharness.)

Providing meaningful timing benchmarks is tricky because, like any new platform, some tests fail, some crash, some hang.

Thankfully, jsreftests generally pass.

Recall that in our current continuous integration, our Android/opt armv7 builds run jsreftests in 40 chunks with a total run time of about 1800 minutes (30 hours!).

From https://bugzilla.mozilla.org/show_bug.cgi?id=1438665#c4, recall that an Android/opt x86 build ran jsreftests on the Talos moonshot hardware in about 100 minutes (18x faster than current armv7).

On a packet.net type 0 instance, I used an Android/opt x86 build to run jsreftests in about 360 minutes (5x faster than current armv7).

On a packet.net type 1 instance, I used an Android/opt x86 build to run jsreftests in about 80 minutes (22x faster than current armv7).

Comment 8

[Geoff Brown [:gbrown]]

Similarly, crashtests pass consistently on all platforms.

Current Android/opt armv7 crashtests: about 120 minutes

Talos moonshot x86: 7 minutes

packet.net type 0 x86: 30 minutes

packet.net type 1 x86: 6 minutes

Comment 9

[Geoff Brown [:gbrown]]

mochitest-chrome has some failures on the new platforms, so maybe it's not a good measure, but...

Current Android/opt armv7 mochitest-chrome: about 100 minutes

Talos moonshot x86: 20 minutes

packet.net type 0 x86: 26 minutes

packet.net type 1 x86: 25 minutes

Comment 10

[Geoff Brown [:gbrown]]

There are lots of timeouts on plain mochitests on the new platforms, so I can't compare them as a whole. However, selecting one large-ish manifest that passes everywhere - dom-level2-html mochitests - I see these numbers:

Current Android/opt armv7: 80 minutes

Talos moonshot x86: 2 minutes

packet.net type 0 x86: 14 minutes

packet.net type 1 x86: 2 minutes

Comment 11

[Geoff Brown [:gbrown]]

In summary, these experiments were very promising:
 
 - it is possible to run an Android 7.0 x86 emulator with kvm acceleration on packet.net type 0 or type 1 servers;
 - performance varies by test type, but type 1 results were similar to moonshot performance (bug 1438665); type 0 results were sometimes about 5x slower, but still considerably faster than our current CI arm emulators.

:coop's team is now trying to provide taskcluster/try access to packet.net (hoping for news in Q2).