1032268 - Experiment with c3.xlarge for Android 2.3 jobs

Status: RESOLVED WORKSFORME

(Infrastructure & Operations Graveyard :: CIDuty, task)

Description

[Geoff Brown [:gbrown]]

+++ This bug was initially created as a clone of Bug #980519 +++

In bug 980519 I compared Android 2.3 test job performance on several different aws instance types. For reftests, it seemed like performance on aws was unacceptable, even on c3.2xlarge. But that was months ago, and there were anomalies in my results, so its worth checking back again.

Today, we are running mochitests, robocop, and xpcshell on aws (m3.medium?) with good success and reasonable performance. 

Let's check particularly on reftest performance on c3.xlarge, since some b2g tests may be running on c3.xlarge soon.

Comment 1

[Geoff Brown [:gbrown]]

Tests on http://ftp.mozilla.org/pub/mozilla.org/mobile/tinderbox-builds/mozilla-central-android/1403886929/fennec-33.0a1.en-US.android-arm.apk, test harness "ran for" times. (See https://tbpl.mozilla.org/?rev=c90b38c47a1d)

       ix (m-c tbpl)      c3.xlarge
R1     0:41:53            0:43:35
R2     0:32:46            0:34:27
R3     0:37:48            0:38:53
R4     0:38:08            0:39:11
R5     0:38:48            0:39:29
R6     0:39:09            0:40:12
R7     0:44:14            0:45:57
R8     0:41:10            0:41:53
R9     0:40:12            0:40:32
R10    0:35:47            0:36:07
R11    0:37:08            0:38:11
R12    0:46:35            0:47:58
R13    0:31:44            0:32:27
R14    0:37:28            0:38:30
R15    0:37:08            0:38:11
R16    0:38:49            0:40:32
       -------            -------
        618:27 (37107s)    636:05 (38165s) = ix+3%

       ix (m-c tbpl)      c3.xlarge
C1     0:27:49            0:31:38
C2     0:23:00            0:25:05

       ix (m-c tbpl)      c3.xlarge
J1     0:27:06            0:29:35
J2     0:33:28            0:36:14
J3     0:28:46            0:31:51
J4     0:31:08            0:33:34
J5     0:29:26            0:31:52
J6     0:28:25            0:31:12

That looks very promising indeed: All reftests seem to run on c3.xlarge just a little slower than on ix. It looks to me like the difference is not large enough to require any new chunks -- we can swap c3.xlarge for ix for all reftests.

All tests ran with 0 unexpected failures and no crashes.

I ran all tests a second time and briefly reviewed the results; I saw no significant differences. But this is just 2 repetitions on a single revision -- we should test more before making production changes.

I cannot account for why these results are significantly better than my previous experiment on c3.2xlarge.

Comment 2

[Geoff Brown [:gbrown]]

Here's a similar analysis for our other Android 2.3 Opt tests, which already run on aws, but on a slower instance type. It was easier to extract "run-tests - Wall time: Xs" here:

       aws (m-c tbpl)     c3.xlarge
M1     1772               691
M2     1322               421
M3     1682               541
M4     2012 *1 fail       541
M5     crash              841
M6     2313               811
M7     1983               781
M8     1502               541
M9     1622               511
M10    1832               631
M11    1382               481
M12    2193               811
Mgl    1865               811

rc1    1982               1261
rc2    1802               1411
rc3    1922               1381
rc4    2103               1382

x1     3604               1592
x2     5045               2372
x3     3063               1532

All of these tests run significantly faster, with mochitests showing the greatest improvement. 

All tests ran without failure on c3.large.

Comment 3

[Geoff Brown [:gbrown]]

Finally, here's a quick comparison of reftests against an armv6 build:

       ix (m-c tbpl)      c3.xlarge
R1     0:45:15            0:46:58
R2     0:35:49            0:37:12
R3     0:41:12            0:42:15
R4     0:41:13            0:42:14

It all seems consistent: reftests run just a little slower on c3.xlarge than on ix.

Comment 4

[Geoff Brown [:gbrown]]

All done here. kmoir is looking at moving more Android 2.3 tests to aws, on c3.xlarge.