Experiment with other instance types for Android 2.3 jobs

RESOLVED FIXED

Status

Release Engineering
Platform Support
RESOLVED FIXED
3 years ago
3 years ago

People

(Reporter: armenzg, Assigned: gbrown)

Tracking

Firefox Tracking Flags

(Not tracked)

Details

(Reporter)

Description

3 years ago
The jobs run rather slow. We want to see what would happen if we run them on faster instances or multi-cores ones.

Let's get you a loaner.

https://tbpl.mozilla.org/?tree=Ash&jobname=Android%202.3

Once we have a better candidate we should write a patch to do the required buildbot changes.

<rail> armenzg: ... we may need to introduce another platform name for this because buildbot is dumb
(Reporter)

Updated

3 years ago
Depends on: 980520
(Assignee)

Updated

3 years ago
Blocks: 967913
(Reporter)

Comment 1

3 years ago
As per email, I've upgraded this instance from m1.medium to c3.2xlarge.

Adding rail for him to be aware with this experiment.
(Assignee)

Comment 2

3 years ago
c3.2xlarge, R1 running via mozharness:

top - 10:27:52 up 1 day,  4:10,  2 users,  load average: 6.10, 4.92, 2.60
Tasks: 171 total,   2 running, 169 sleeping,   0 stopped,   0 zombie
Cpu(s): 44.2%us,  1.2%sy,  0.0%ni, 54.5%id,  0.0%wa,  0.0%hi,  0.0%si,  0.1%st
Mem:  15339148k total,  5815076k used,  9524072k free,    77884k buffers
Swap:        0k total,        0k used,        0k free,  4089484k cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND            
 2575 cltbld    20   0 3484m 896m  22m R  360  6.0  24:42.95 emulator64-arm     
  746 cltbld    20   0  291m 129m  26m S   12  0.9   0:49.26 Xvfb               
 1186 cltbld    20   0  346m  12m 9868 S    1  0.1   0:06.54 metacity           
 2723 cltbld    20   0 17340 1344  960 R    0  0.0   0:00.15 top                

----------

top - 11:13:07 up 1 day,  4:55,  2 users,  load average: 4.32, 4.72, 4.86
Tasks: 171 total,   2 running, 169 sleeping,   0 stopped,   0 zombie
Cpu(s): 56.4%us,  1.8%sy,  0.0%ni, 41.6%id,  0.0%wa,  0.0%hi,  0.0%si,  0.2%st
Mem:  15339148k total,  5872012k used,  9467136k free,    81316k buffers
Swap:        0k total,        0k used,        0k free,  4089844k cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND            
 2575 cltbld    20   0 3556m 932m  22m R  454  6.2 199:06.16 emulator64-arm     
  746 cltbld    20   0  291m 129m  26m S   16  0.9   6:37.48 Xvfb               
 1186 cltbld    20   0  346m  12m 9868 S    2  0.1   0:51.47 metacity           
   26 root      20   0     0    0    0 S    0  0.0   0:00.19 ksoftirqd/5        

--------------

11:22:24     INFO -  REFTEST TEST-LOAD | http://10.0.2.2:8854/tests/image/test/reftest/apng/bug411852-1-ref.png | 507 / 3554 (14%)
11:22:24     INFO -  REFTEST TEST-KNOWN-FAIL(EXPECTED RANDOM) | http://10.0.2.2:8854/tests/image/test/reftest/apng/delaytest.html?bug411852-1.png | image comparison (==)
11:22:24     INFO -  REFTEST INFO | Loading a blank page
11:22:24     INFO -  REFTEST TEST-END | http://10.0.2.2:8854/tests/image/test/reftest/apng/delaytest.html?bug411852-1.png
11:22:24     INFO -  REFTEST TEST-START | http://10.0.2.2:8854/tests/image/test/reftest/apng/delaytest.html?bug546272.png
11:22:24     INFO -  REFTEST TEST-LOAD | http://10.0.2.2:8854/tests/image/test/reftest/apng/delaytest.html?bug546272.png | 508 / 3554 (14%)
11:22:24  WARNING -  TEST-UNEXPECTED-FAIL | http://10.0.2.2:8854/tests/image/test/reftest/apng/delaytest.html?bug546272.png | application ran for longer than allowed maximum time
11:22:24     INFO -  INFO | automation.py | Application ran for: 1:00:51.307762

11:22:25     INFO - Total resource usage - Wall time: 3715s; CPU: 48.0%; Read bytes: 1245184; Write bytes: 294600704; Read time: 572; Write time: 1276600
11:22:25     INFO - install - Wall time: 22s; CPU: 13.0%; Read bytes: 0; Write bytes: 7294976; Read time: 0; Write time: 17276
11:22:25     INFO - run-tests - Wall time: 3693s; CPU: 48.0%; Read bytes: 1040384; Write bytes: 286666752; Read time: 496; Write time: 1259012
11:22:25     INFO - stop-emulators - Wall time: 0s; CPU: Can't collect data; Read bytes: 0; Write bytes: 0; Read time: 0; Write time: 0
(Assignee)

Comment 3

3 years ago
top - 11:36:42 up 1 day,  5:19,  2 users,  load average: 3.56, 3.34, 3.45
Tasks: 172 total,   2 running, 170 sleeping,   0 stopped,   0 zombie
Cpu0  : 43.3%us,  1.4%sy,  0.0%ni, 55.3%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Cpu1  : 41.6%us,  0.7%sy,  0.0%ni, 57.3%id,  0.0%wa,  0.0%hi,  0.0%si,  0.3%st
Cpu2  : 43.1%us,  0.0%sy,  0.0%ni, 56.6%id,  0.0%wa,  0.0%hi,  0.0%si,  0.3%st
Cpu3  : 52.7%us,  1.4%sy,  0.0%ni, 45.5%id,  0.0%wa,  0.0%hi,  0.0%si,  0.3%st
Cpu4  : 58.2%us,  1.4%sy,  0.0%ni, 40.1%id,  0.0%wa,  0.0%hi,  0.0%si,  0.3%st
Cpu5  : 42.5%us,  1.7%sy,  0.0%ni, 55.4%id,  0.0%wa,  0.0%hi,  0.0%si,  0.3%st
Cpu6  : 39.9%us,  0.7%sy,  0.0%ni, 59.4%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Cpu7  : 41.6%us,  1.4%sy,  0.0%ni, 56.7%id,  0.0%wa,  0.0%hi,  0.0%si,  0.3%st
Mem:  15339148k total,  5874052k used,  9465096k free,    83104k buffers
Swap:        0k total,        0k used,        0k free,  4207028k cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND            
 2784 cltbld    20   0 3483m 833m  22m R  370  5.6   9:07.16 emulator64-arm     
  746 cltbld    20   0  291m 130m  27m S   12  0.9   8:08.47 Xvfb               
 1186 cltbld    20   0  346m  12m 9868 S    2  0.1   1:03.25 metacity           
 1259 cltbld     9 -11  271m 4280 3048 S    0  0.0   0:05.91 pulseaudio         
 2822 cltbld    20   0 17340 1352  960 R    0  0.0   0:00.68 top
(Assignee)

Comment 4

3 years ago
For comparison, an R3 run on c3.2xlarge:

12:35:23     INFO -  REFTEST TEST-PASS | http://10.0.2.2:8854/tests/layout/reftests/position-dynamic-changes/vertical/toauto-topA-heightN-bottomN.html?padding_parent | image comparison (==)
12:35:23     INFO -  REFTEST INFO | Loading a blank page
12:35:23     INFO -  REFTEST TEST-END | http://10.0.2.2:8854/tests/layout/reftests/position-dynamic-changes/vertical/toauto-topA-heightN-bottomN.html?padding_parent
12:35:23     INFO -  REFTEST TEST-START | http://10.0.2.2:8854/tests/layout/reftests/position-dynamic-changes/vertical/toauto-topA-heightA-bottomN.html?padding_parent
12:35:23     INFO -  REFTEST TEST-LOAD | http://10.0.2.2:8854/tests/layout/reftests/position-dynamic-changes/vertical/toauto-topA-heightA-bottomN.html?padding_parent | 81 / 3189 (2%)
12:35:23  WARNING -  TEST-UNEXPECTED-FAIL | http://10.0.2.2:8854/tests/layout/reftests/position-dynamic-changes/vertical/toauto-topA-heightA-bottomN.html?padding_parent | application ran for longer than allowed maximum time
12:35:23     INFO -  INFO | automation.py | Application ran for: 1:00:46.737086
(Assignee)

Comment 5

3 years ago
jsreftests on c3.2xlarge:

13:51:18     INFO -  REFTEST TEST-LOAD | http://10.0.2.2:8854/jsreftest/tests/jsreftest.html?test=js1_5/Regress/regress-360969-05.js | 1932 / 6734 (28%)
13:51:18  WARNING -  TEST-UNEXPECTED-FAIL | http://10.0.2.2:8854/jsreftest/tests/jsreftest.html?test=js1_5/Regress/regress-360969-05.js | application ran for longer than allowed maximum time
13:51:18     INFO -  INFO | automation.py | Application ran for: 1:01:20.542796

13:51:19     INFO - Total resource usage - Wall time: 3760s; CPU: 51.0%; Read bytes: 0; Write bytes: 406016000; Read time: 0; Write time: 1557784
13:51:19     INFO - install - Wall time: 21s; CPU: 13.0%; Read bytes: 0; Write bytes: 61440; Read time: 0; Write time: 8
13:51:19     INFO - run-tests - Wall time: 3738s; CPU: 52.0%; Read bytes: 0; Write bytes: 398024704; Read time: 0; Write time: 1534840
13:51:19     INFO - stop-emulators - Wall time: 0s; CPU: Can't collect data; Read bytes: 0; Write bytes: 0; Read time: 0; Write time: 0

top - 14:25:44 up 1 day,  8:08,  2 users,  load average: 7.73, 4.58, 2.82
Tasks: 172 total,   2 running, 170 sleeping,   0 stopped,   0 zombie
Cpu0  : 47.6%us,  0.0%sy,  0.0%ni, 52.0%id,  0.3%wa,  0.0%hi,  0.0%si,  0.0%st
Cpu1  : 47.6%us,  1.4%sy,  0.0%ni, 51.0%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Cpu2  : 47.6%us,  1.4%sy,  0.0%ni, 50.7%id,  0.0%wa,  0.0%hi,  0.0%si,  0.3%st
Cpu3  : 98.7%us,  1.3%sy,  0.0%ni,  0.0%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Cpu4  : 46.3%us,  1.0%sy,  0.0%ni, 52.4%id,  0.0%wa,  0.3%hi,  0.0%si,  0.0%st
Cpu5  : 48.0%us,  0.7%sy,  0.0%ni, 51.4%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Cpu6  : 47.4%us,  0.7%sy,  0.0%ni, 51.9%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Cpu7  : 47.1%us,  1.0%sy,  0.0%ni, 51.9%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Mem:  15339148k total,  6141012k used,  9198136k free,    93296k buffers
Swap:        0k total,        0k used,        0k free,  4443352k cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND            
 3156 cltbld    20   0 3484m 841m  22m R  426  5.6  15:59.61 emulator64-arm     
  746 cltbld    20   0  291m 130m  27m S   14  0.9  25:22.69 Xvfb               
 3270 cltbld    20   0  357m  33m  18m S    3  0.2   0:05.75 xpcshell           
 1186 cltbld    20   0  346m  12m 9868 S    2  0.1   3:16.35 metacity           
 1259 cltbld     9 -11  271m 4280 3048 S    0  0.0   0:17.19 pulseaudio         
 3011 cltbld    20   0 17340 1352  960 R    0  0.0   0:12.33 top                
 3254 cltbld    20   0 61896  11m 1096 S    0  0.1   0:00.23 python
(Assignee)

Comment 6

3 years ago
M1 on c3.2xlarge:

15:48:02     INFO -  1274 INFO Passed:  31197
15:48:02     INFO -  1275 INFO Failed:  0
15:48:02     INFO -  1276 INFO Todo:    63
15:48:02     INFO -  1277 INFO Slowest: 35209ms - /tests/content/base/test/test_bug682592.html
15:48:02     INFO -  1278 INFO SimpleTest FINISHED
15:48:02     INFO -  
15:48:02     INFO -  INFO | automation.py | Application ran for: 0:10:28.016660

vs. normal m3.medium (ash):

09:23:09     INFO -  1274 INFO Passed:  31203
09:23:09     INFO -  1275 INFO Failed:  0
09:23:09     INFO -  1276 INFO Todo:    63
09:23:09     INFO -  1277 INFO Slowest: 209635ms - /tests/content/base/test/test_bug682592.html
09:23:09     INFO -  1278 INFO SimpleTest FINISHED
09:23:09     INFO -  INFO | automation.py | Application ran for: 0:35:41.721845
(Reporter)

Comment 7

3 years ago
Changed now to be a m3.large.
(Assignee)

Comment 8

3 years ago
m3.large, R1 running via mozharness:

top - 09:35:31 up  2:46,  1 user,  load average: 2.36, 1.36, 0.67
Tasks: 139 total,   2 running, 137 sleeping,   0 stopped,   0 zombie
Cpu0  : 92.0%us,  4.3%sy,  0.0%ni,  3.7%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Cpu1  : 91.7%us,  5.3%sy,  0.0%ni,  3.0%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Mem:   7629428k total,  5576772k used,  2052656k free,   147680k buffers
Swap:        0k total,        0k used,        0k free,  4079108k cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+ COMMAND 
 1629 cltbld    20   0 3139m 710m  22m R  178  9.5   2:40.44 emulator64-arm
  659 cltbld    20   0  242m 122m  27m S    8  1.6   0:06.45 Xvfb  
 1741 cltbld    20   0  357m  33m  18m S    6  0.4   0:00.97 xpcshell  
 1105 cltbld    20   0  346m  12m 9868 S    1  0.2   0:00.97 metacity       
 1627 cltbld    20   0 17340 1312  960 R    0  0.0   0:00.81 top 

-----------------

top - 09:44:31 up  2:55,  1 user,  load average: 2.80, 2.50, 1.57
Tasks: 139 total,   2 running, 137 sleeping,   0 stopped,   0 zombie
Cpu0  : 86.3%us,  6.5%sy,  0.0%ni,  7.2%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Cpu1  : 86.3%us,  5.5%sy,  0.0%ni,  8.2%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Mem:   7629428k total,  5755440k used,  1873988k free,   147684k buffers
Swap:        0k total,        0k used,        0k free,  4079344k cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
 1629 cltbld    20   0 3139m 883m  22m R  174 11.9  18:04.15 emulator64-arm      
  659 cltbld    20   0  242m 122m  27m S    8  1.6   0:48.20 Xvfb   
 1105 cltbld    20   0  346m  12m 9868 S    1  0.2   0:07.57 metacity   
 1725 cltbld    20   0 61780  11m 1092 S    0  0.2   0:00.58 python       
    1 root      20   0 24332 2316 1364 S    0  0.0   0:00.39 init

-----------------

10:36:08     INFO -  REFTEST TEST-LOAD | http://10.0.2.2:8854/tests/image/test/reftest/pngsuite-oddsizes/s39i3p04.png | 241 / 3554 (6%)
10:36:08     INFO -  REFTEST TEST-LOAD | http://10.0.2.2:8854/tests/image/test/reftest/pngsuite-oddsizes/s39_3p04.html | 241 / 3554 (6%)
10:36:08     INFO -  REFTEST TEST-KNOWN-FAIL | http://10.0.2.2:8854/tests/image/test/reftest/pngsuite-oddsizes/s39i3p04.png | image comparison (==)
10:36:08     INFO -  REFTEST INFO | Loading a blank page 
10:36:08  WARNING -  TEST-UNEXPECTED-FAIL | http://10.0.2.2:8854/tests/image/test/reftest/pngsuite-oddsizes/s39i3p04.png | application ran for longer than allowed maximum time
10:36:08     INFO -  INFO | automation.py | Application ran for: 1:01:07.094051

10:36:10     INFO - Total resource usage - Wall time: 3717s; CPU: 88.0%; Read bytes: 1302528; Write bytes: 294555648; Read time: 936; Write time: 1137124
10:36:10     INFO - install - Wall time: 24s; CPU: 58.0%; Read bytes: 16384; Write bytes: 79720448; Read time: 20; Write time: 395928
10:36:10     INFO - run-tests - Wall time: 3693s; CPU: 89.0%; Read bytes: 1097728; Write bytes: 198160384; Read time: 504; Write time: 636604
10:36:10     INFO - stop-emulators - Wall time: 0s; CPU: Can't collect data; Read bytes: 0; Write bytes: 0; Read time: 0; Write time: 0
(Assignee)

Comment 9

3 years ago
Now m3.medium.
(Reporter)

Comment 10

3 years ago
From https://bugzilla.mozilla.org/show_bug.cgi?id=980520#c3 it seems that you don't need the machine again.

Did we significantly better results on other instances?
(Assignee)

Comment 11

3 years ago
m3.medium, R1 running via mozharness:

top - 15:12:35 up 1 day,  9:50,  1 user,  load average: 1.38, 1.17, 0.60
Tasks: 135 total,   2 running, 133 sleeping,   0 stopped,   0 zombie
Cpu0  : 36.6%us,  6.0%sy,  0.0%ni, 24.4%id,  4.0%wa,  0.0%hi,  0.1%si, 28.9%st
Mem:   3840420k total,  3821444k used,    18976k free,    91756k buffers
Swap:        0k total,        0k used,        0k free,  2678332k cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
 8041 cltbld    20   0 2652m 474m  22m R 91.3 12.7   2:30.51 emulator64-arm
  664 cltbld    20   0  226m 122m  26m S  1.0  3.3  45:54.20 Xvfb
 1014 cltbld    20   0 95880  10m 1880 S  1.0  0.3  10:33.58 x11vnc
 1169 cltbld     9 -11  271m 3984 2696 S  0.7  0.1  11:59.44 pulseaudio
 8141 cltbld    20   0 66468  16m 4444 S  0.7  0.4   0:00.48 python

----------------

08:37:55     INFO -  REFTEST TEST-LOAD | http://10.0.2.2:8854/tests/image/test/reftest/pngsuite-ancillary/cdsn2c08.png | 146 / 3554 (4%)
08:37:55     INFO -
08:37:55  WARNING -  TEST-UNEXPECTED-FAIL | http://10.0.2.2:8854/tests/image/test/reftest/pngsuite-ancillary/cdsn2c08.png | application ran for longer than allowed maximum time
08:37:55     INFO -  INFO | automation.py | Application ran for: 1:02:14.502592

08:37:57     INFO - Total resource usage - Wall time: 3843s; CPU: 99.0%; Read bytes: 17743872; Write bytes: 307716096; Read time: 6700; Write time: 1104932
08:37:57     INFO - install - Wall time: 59s; CPU: 100.0%; Read bytes: 0; Write bytes: 197644288; Read time: 0; Write time: 880212
08:37:57     INFO - run-tests - Wall time: 3784s; CPU: 99.0%; Read bytes: 17526784; Write bytes: 109764608; Read time: 6584; Write time: 224648
(Assignee)

Comment 12

3 years ago
R1 via mozharness on my ix loaner:

14:13:03     INFO -  REFTEST TEST-LOAD | http://10.0.2.2:8854/tests/layout/reftests/w3c-css/submitted/variables/variable-supports-62.html | 1058 / 3554 (29%)
14:13:03     INFO -  REFTEST TEST-LOAD | http://10.0.2.2:8854/tests/layout/reftests/w3c-css/submitted/variables/support/color-green-ref.html | 1058 / 3554 (29%)
14:13:03  WARNING -  TEST-UNEXPECTED-FAIL | http://10.0.2.2:8854/tests/layout/reftests/w3c-css/submitted/variables/variable-supports-62.html | application ran for longer than allowed maximum time
14:13:03     INFO -  INFO | automation.py | Application ran for: 1:00:55.757441

14:13:04     INFO - Total resource usage - Wall time: 3715s; CPU: 13.0%; Read bytes: 8192; Write bytes: 522919936; Read time: 464; Write time: 716564
14:13:04     INFO - install - Wall time: 22s; CPU: 13.0%; Read bytes: 0; Write bytes: 78622720; Read time: 0; Write time: 46476
14:13:04     INFO - run-tests - Wall time: 3694s; CPU: 13.0%; Read bytes: 8192; Write bytes: 441323520; Read time: 464; Write time: 669928
(Assignee)

Comment 13

3 years ago
R1 via remotereftest.py on my w540 laptop:

REFTEST TEST-LOAD | http://10.0.2.2:8854/tests/layout/reftests/bidi/unicode-bidi-plaintext-textarea-2.html | 1626 / 3554 (45%)
TEST-UNEXPECTED-FAIL | http://10.0.2.2:8854/tests/layout/reftests/bidi/unicode-bidi-plaintext-textarea-2.html | application ran for longer than allowed maximum time
INFO | automation.py | Application ran for: 1:00:47.541895
(Assignee)

Comment 14

3 years ago
Let's summarize these results in terms of "how long would it take to run all plain-reftests on each platform?" This will be a gross approximation: If R1 was 10% complete after 1 hour, I'll assume it would take 10 hours to complete R1, and an additional 10 hours to complete each of R2 and R3, for a total of 30 hours.

m3.medium    75 hours
m3.large     50 hours
c3.2xlarge   22 hours
ix           11 hours
w540          7 hours

We cannot use my laptop -- that's just an interesting result.

We could use ix slaves. Allowing for some variation in per-chunk run times, I imagine we could run reftests in about 16 chunks.

We probably cannot reasonably use aws for reftests (unless profiling/optimization improves these results dramatically).
(Assignee)

Comment 15

3 years ago
Let's also look at mochitests, comparing a single run of each chunk on m3.medium to one on m3.large:

       m3.medium      m3.large
M1       0:37:00      0:12:50
M2       0:32:39      0:10:29
M3       0:35:32      0:11:11
M4       1:02:09      0:22:44
M5       1:01:20      0:20:41
M6       0:44:50      0:15:52
M7       0:42:23      0:13:53
M8       0:31:01      0:10:11

The m3.large run times are about 1/3 of the m3.medium times and I understand that m3.large costs about 2x m3.medium -- it seems to me that running these on m3.large would reduce our overall costs.

Notice the top results (comments 8 and 11 especially): the emulator fully utilizes the cpus on m3.large, but does not on m3.medium (less memory?).

Also, most of the tests (mochitests, crashtests, robocop) disabled for 2.3 are caused by timeouts, or show evidence of Android services not keeping up with requests. I wonder if these problems would be lessened on m3.large.

So although a "bigger" aws instance type will not help us run reftests, I think it would be better to run the remaining tests on m3.large. Armen (or appropriate rel-eng proxy) -- can we run Android 2.3 tests on m3.large?
Flags: needinfo?(armenzg)
(Reporter)

Comment 16

3 years ago
(adding kmoir so she can be in the loop once she's back from EclispeCon)

Running on a different instance type than our regular test EC2 might not be an easy task.
rail: what would it be involved?

gbrown:
* I'm happy to run the reftests on the Linux ix machines since it is an under-utilized pool of machines.
* m3.large seems a better choice than m3.medium

What are the m1.medium wall time numbers? (Our current setup)

Do we know what is the reason for the slow jobs?
Have any of the numbers improved after we deployed a newer version of the Mesa libs? (2 weeks ago)
Flags: needinfo?(armenzg) → needinfo?(rail)
(Assignee)

Comment 17

3 years ago
(In reply to Armen Zambrano [:armenzg] (Release Engineering) (EDT/UTC-4) from comment #16)
> (adding kmoir so she can be in the loop once she's back from EclispeCon)
> What are the m1.medium wall time numbers? (Our current setup)

Oh, are we running m1.medium currently? I thought they were m3.medium.

Anyway, the wall time numbers on ash seem very close to my m3.medium observations.
 
> Do we know what is the reason for the slow jobs?

No. I am hoping to find time to profile reftests again (dminor did earlier, but that did not point to a solution and we never followed up).

> Have any of the numbers improved after we deployed a newer version of the
> Mesa libs? (2 weeks ago)

No. All the current times seem about the same as they were at the end of February.
(In reply to Armen Zambrano [:armenzg] (Release Engineering) (EDT/UTC-4) from comment #16)
> (adding kmoir so she can be in the loop once she's back from EclispeCon)
> 
> Running on a different instance type than our regular test EC2 might not be
> an easy task.
> rail: what would it be involved?

* Buildbot has no idea about the instance types we use. We'll need different naming for these instances.
* Changing the instance type for all slaves may be problematic, see bug 969590.
Flags: needinfo?(rail)
(Reporter)

Updated

3 years ago
Blocks: 985650
(Assignee)

Comment 19

3 years ago
(In reply to Geoff Brown [:gbrown] from comment #14)
> Let's summarize these results in terms of "how long would it take to run all
> plain-reftests on each platform?" 
> 
> m3.medium    75 hours
> m3.large     50 hours
> c3.2xlarge   22 hours
> ix           11 hours
> w540          7 hours

Between the March 13 and March 18 merges to Ash, there was a dramatic improvement in reftest performance. Ash run-times which were equivalent to m3.medium now run 3 to 4 times faster and ix runs about twice as fast:

Ash (like m3.medium?) 24 hours
ix                     6 hours
(Assignee)

Updated

3 years ago
Status: NEW → RESOLVED
Last Resolved: 3 years ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.