Closed Bug 1032268 Opened 11 years ago Closed 11 years ago

Experiment with c3.xlarge for Android 2.3 jobs

Categories

(Infrastructure & Operations Graveyard :: CIDuty, task)

Product:
Infrastructure & Operations Graveyard
Component:
CIDuty
Platform:
x86_64
Linux
Type:
task
Priority:
Not set
Severity:
normal

Tracking

(Not tracked)

Status:
RESOLVED WORKSFORME

People

(Reporter: gbrown, Assigned: gbrown)

References

Details

+++ This bug was initially created as a clone of Bug #980519 +++ In bug 980519 I compared Android 2.3 test job performance on several different aws instance types. For reftests, it seemed like performance on aws was unacceptable, even on c3.2xlarge. But that was months ago, and there were anomalies in my results, so its worth checking back again. Today, we are running mochitests, robocop, and xpcshell on aws (m3.medium?) with good success and reasonable performance. Let's check particularly on reftest performance on c3.xlarge, since some b2g tests may be running on c3.xlarge soon.
Tests on http://ftp.mozilla.org/pub/mozilla.org/mobile/tinderbox-builds/mozilla-central-android/1403886929/fennec-33.0a1.en-US.android-arm.apk, test harness "ran for" times. (See https://tbpl.mozilla.org/?rev=c90b38c47a1d) ix (m-c tbpl) c3.xlarge R1 0:41:53 0:43:35 R2 0:32:46 0:34:27 R3 0:37:48 0:38:53 R4 0:38:08 0:39:11 R5 0:38:48 0:39:29 R6 0:39:09 0:40:12 R7 0:44:14 0:45:57 R8 0:41:10 0:41:53 R9 0:40:12 0:40:32 R10 0:35:47 0:36:07 R11 0:37:08 0:38:11 R12 0:46:35 0:47:58 R13 0:31:44 0:32:27 R14 0:37:28 0:38:30 R15 0:37:08 0:38:11 R16 0:38:49 0:40:32 ------- ------- 618:27 (37107s) 636:05 (38165s) = ix+3% ix (m-c tbpl) c3.xlarge C1 0:27:49 0:31:38 C2 0:23:00 0:25:05 ix (m-c tbpl) c3.xlarge J1 0:27:06 0:29:35 J2 0:33:28 0:36:14 J3 0:28:46 0:31:51 J4 0:31:08 0:33:34 J5 0:29:26 0:31:52 J6 0:28:25 0:31:12 That looks very promising indeed: All reftests seem to run on c3.xlarge just a little slower than on ix. It looks to me like the difference is not large enough to require any new chunks -- we can swap c3.xlarge for ix for all reftests. All tests ran with 0 unexpected failures and no crashes. I ran all tests a second time and briefly reviewed the results; I saw no significant differences. But this is just 2 repetitions on a single revision -- we should test more before making production changes. I cannot account for why these results are significantly better than my previous experiment on c3.2xlarge.
Here's a similar analysis for our other Android 2.3 Opt tests, which already run on aws, but on a slower instance type. It was easier to extract "run-tests - Wall time: Xs" here: aws (m-c tbpl) c3.xlarge M1 1772 691 M2 1322 421 M3 1682 541 M4 2012 *1 fail 541 M5 crash 841 M6 2313 811 M7 1983 781 M8 1502 541 M9 1622 511 M10 1832 631 M11 1382 481 M12 2193 811 Mgl 1865 811 rc1 1982 1261 rc2 1802 1411 rc3 1922 1381 rc4 2103 1382 x1 3604 1592 x2 5045 2372 x3 3063 1532 All of these tests run significantly faster, with mochitests showing the greatest improvement. All tests ran without failure on c3.large.
Finally, here's a quick comparison of reftests against an armv6 build: ix (m-c tbpl) c3.xlarge R1 0:45:15 0:46:58 R2 0:35:49 0:37:12 R3 0:41:12 0:42:15 R4 0:41:13 0:42:14 It all seems consistent: reftests run just a little slower on c3.xlarge than on ix.
All done here. kmoir is looking at moving more Android 2.3 tests to aws, on c3.xlarge.
Status: NEW → RESOLVED
Closed: 11 years ago
Resolution: --- → WORKSFORME
Component: Platform Support → Buildduty
Product: Release Engineering → Infrastructure & Operations
Product: Infrastructure & Operations → Infrastructure & Operations Graveyard
You need to log in before you can comment on or make changes to this bug.