967913 - Android 2.3 reftests are too slow

Status: RESOLVED FIXED

(Testing :: General, defect)

Description

[Geoff Brown [:gbrown]]

The first results for running reftests on Android 2.3 emulators show reftests running very slowly.

https://tbpl.mozilla.org/php/getParsedLog.php?id=34039984&tree=Ash#error0

19:53:29     INFO -  REFTEST TEST-LOAD | http://10.0.2.2:8854/tests/image/test/reftest/pngsuite-ancillary/ch2n3p08.png | 149 / 3451 (4%)
19:53:29     INFO -  REFTEST TEST-LOAD | http://10.0.2.2:8854/tests/image/test/reftest/pngsuite-ancillary/ch2n3p08.html | 149 / 3451 (4%)
19:53:29  WARNING -  TEST-UNEXPECTED-FAIL | http://10.0.2.2:8854/tests/image/test/reftest/pngsuite-ancillary/ch2n3p08.png | application ran for longer than allowed maximum time
19:53:29     INFO -  INFO | automation.py | Application ran for: 1:02:20.784677

These are running in 3 chunks already, so 4% is really about 1% of the total, running in 1 hour, suggesting we need nearly 100 hours to run all of the (plain) reftests.

jsreftests are similar:

19:47:30     INFO -  REFTEST TEST-END | http://10.0.2.2:8854/jsreftest/tests/jsreftest.html?test=ecma/LexicalConventions/7.5-3-n.js
19:47:30     INFO -  REFTEST TEST-START | http://10.0.2.2:8854/jsreftest/tests/jsreftest.html?test=ecma/LexicalConventions/7.5-4-n.js
19:47:31     INFO -  REFTEST TEST-LOAD | http://10.0.2.2:8854/jsreftest/tests/jsreftest.html?test=ecma/LexicalConventions/7.5-4-n.js | 398 / 6714 (5%)
19:47:31  WARNING -  TEST-UNEXPECTED-FAIL | http://10.0.2.2:8854/jsreftest/tests/jsreftest.html?test=ecma/LexicalConventions/7.5-4-n.js | application ran for longer than allowed maximum time
19:47:31     INFO -  INFO | automation.py | Application ran for: 1:03:37.009765

Crashtests are slow but could be run in perhaps 4 chunks:

9:46:13     INFO -  REFTEST INFO | Loading a blank page
19:46:13     INFO -  REFTEST TEST-END | http://10.0.2.2:8854/tests/js/xpconnect/crashtests/117307-1.html
19:46:13     INFO -  REFTEST TEST-START | http://10.0.2.2:8854/tests/js/xpconnect/crashtests/193710.html
19:46:13     INFO -  REFTEST TEST-LOAD | http://10.0.2.2:8854/tests/js/xpconnect/crashtests/193710.html | 719 / 2608 (27%)
19:46:13  WARNING -  TEST-UNEXPECTED-FAIL | http://10.0.2.2:8854/tests/js/xpconnect/crashtests/193710.html | application ran for longer than allowed maximum time
19:46:13     INFO -  INFO | automation.py | Application ran for: 1:03:40.779075


We had a similar issue for Android x86 tests on emulator, which we resolved by using kvm. I assume our acceleration options are more limited for Android 2.3 on ec2.

Comment 1

[Dan Minor [:dminor]]

I've spent some time investigating this.

Using the Android profiler (am profile) it seems that the we are getting bogged down in the layout code, e.g.
    ViewRoot.handleMessage, ViewRoot.measureTraversals (27.3%)
    View.measure, View.layout
    ViewGroup.measureChildWithMargins
    FrameLayout.onMeasure
    RelativeLayout.onMeasure, RelativeLayout.measureChildHorizontal, RelativeLayout.measureChild
    FrameLayout.onLayout
    and a bunch of other layout stuff

I then spent some time attempting to get an SPS profile without much success.

Running the emulators on ec2 I had difficulties connecting to the remote debugger with either the GeckoProfiler plugin or just telnet. Even once connected I was unable to retrieve a profile.

I tried running the emulator locally. I was able to connect and retrieve a profile there, but fennec would crash consistently after 15 or 20 seconds of the profiler being connected.

The profile I got was not consistent with what I saw under ec2 using the Android profiler, probably because of the short period I profiled or the fact that the emulator runs much better locally than under ec2.

Running the emulator locally is around 8x faster than running it under ec2, so it is likely not CPU bound in the same way that is under ec2.

Comment 2

[Dan Minor [:dminor]]

My test instance was restarted as a c3.large instead of a m1.medium and I reran the reftests.

Sadly, the tests were only 50% faster (6% of chunk 1 instead of 4%) in this configuration, so I don't think different AWS instances are going to gain us much.

Comment 3

[Dan Minor [:dminor]]

I recompiled the emulator with gprof support (-pg). Unfortunately it crashes shortly after startup and well before I can get a useful reftest profile from it.

I've been asked to work on other things, so I'm unassigning myself.

Comment 4

[Dan Minor [:dminor]]

The android profile is located here: http://people.mozilla.org/~dminor/reftest-trace.txt
The sps profile is located here: http://people.mozilla.org/~dminor/fennec-reftest-profile.bin

Comment 5

[Geoff Brown [:gbrown]]

Bug 991279 has some observations on Android 2.3 performance which might be of interest here (but 991279 is about an xpcshell test, where the httpd server is running on the emulator).

Comment 6

[Geoff Brown [:gbrown]]

We began running Android 2.3 tests on Ash using ix slaves, on April 20. Run-times are much improved, as anticipated. 

js-reftests run to completion in 6 chunks, each taking 40-45 minutes; this seems about right. 

crashtests run to completion in 5 chunks, each taking 14-22 minutes; we can reduce this to 3 chunks probably.

Initially (April 20), plain reftests ran to completion in 10 chunks, each taking 45-52 minutes. In a new merge today, most chunks take close to 60 minutes, with R5 timing out near the end of the job; I'm hoping that an increase to 12 chunks will make this more reliable.

Comment 7

[Geoff Brown [:gbrown]]

Attachment: adjust chunks for ix - mozharness

See Comment 6. 

This should reduce the crashtests chunks to 3, and increase the plain reftest chunks to 12.

Comment 8

[Geoff Brown [:gbrown]]

Attachment: adjust chunks for ix - buildbot

Comment 9

[Geoff Brown [:gbrown]]

https://hg.mozilla.org/build/mozharness/rev/b8bc238aa1a8
https://hg.mozilla.org/build/buildbot-configs/rev/ac7f8b0a69fd

Comment 10

[Geoff Brown [:gbrown]]

Verified on ash: 
 - crashtests in 3 chunks on ix run fine still: max run time is about 25 minutes
 - plain reftests in 12 chunks on ix now run without any timeouts: max run time is about 55 minutes