944113 - s/m3.xlarge/c3.xlarge/

Status: RESOLVED FIXED

(Infrastructure & Operations Graveyard :: CIDuty, task)

Description

[(dormant account)]

c3 have 1 more ecu, faster IO, faster networking, 50% less ram and costs 33% less. Overall we should be same speed or faster for less $$$. I asked glandium to doublecheck.

In general newer c* nodes seem to offer better value. We should be switching from m* to c* nodes.

Comment 1

[Mike Hommey [:glandium]]

I've set up a c3.xlarge and a m3.xlarge instances with the same OS (Ubuntu LTS, fwiw), and built current m-c on it.
Build times are identical on both when the mozilla-central clone and build objdir are on EBS (around 26:30, without ccache). But c3.xlarge also has internal storage (according to the description, they are SSD), and placing the m-c clone and the build objdir there instead of EBS makes the build one minute faster.

Comment 2

[Gregory Szorc [:gps]]

Should we not invest in the c3.2x, c3.4x, or even c3.8x? The price is linear with number of cores and the build system can utilize those cores. This would, of course, reduce turnaround times for checkins, resulting in faster try pushes, shorter tree closures, etc. The biggest issue I see is that a "build" job consists of not only the highly-concurrent building process, but a lot of single or low core tasks, such as packaging and |make check|. These CPU inefficient tasks would effectively waste money on a high core machine. Of course, we can look into offloading those tasks to other builders (e.g. bug 822394) or improving the jobs to do more of the post-build tasks in parallel.

Comment 3

[(dormant account)]

(In reply to Gregory Szorc [:gps] from comment #2)
> Should we not invest in the c3.2x, c3.4x, or even c3.8x? The price is linear
> with number of cores and the build system can utilize those cores. This
> would, of course, reduce turnaround times for checkins, resulting in faster
> try pushes, shorter tree closures, etc. The biggest issue I see is that a
> "build" job consists of not only the highly-concurrent building process, but
> a lot of single or low core tasks, such as packaging and |make check|. These
> CPU inefficient tasks would effectively waste money on a high core machine.
> Of course, we can look into offloading those tasks to other builders (e.g.
> bug 822394) or improving the jobs to do more of the post-build tasks in
> parallel.

No. This bug is about doing a no-brainer, straight-forward switch to save $$$. Using a bigger instance effectively(keeping 32 cores idle is big opportunity cost) will require some significant infra rework. 

Mike ran some windows builds on diff c3* nodes and a) their raw cpu perf doesn't seem to scale linearly b) cost goes up 
13:37 <taras> so $0.180 for c3.xlarge, $0.2 FOR c3.2xlarge and $0.32 for the other
based on
13:34 <glandium> fwiw, a windows with mozmake on c3.xlarge takes ~35min, on c3.2xlarge, ~20min and on c3.4xlarge, ~16min

I'd like to work on moving to the fastest cpu nodes for building in the medium-term future, but for this year I'd simply like to simply make sure that we are paying what we should for infra we have now.

Comment 4

[Rail Aliiev [:rail]]

I'm going to change the instance type for some builder right now and watch them this week. If we are satisfied with the results we can switch to c3.xlarge for all builders.

Comment 5

[Rail Aliiev [:rail]]

Try:

try-linux64-ec2-002
try-linux64-ec2-004
try-linux64-ec2-005
try-linux64-ec2-007
try-linux64-ec2-011
try-linux64-ec2-015
try-linux64-ec2-020
try-linux64-ec2-023
try-linux64-ec2-028
try-linux64-ec2-031

Comment 6

[Rail Aliiev [:rail]]

Production builders:

bld-linux64-ec2-003
bld-linux64-ec2-004
bld-linux64-ec2-012
bld-linux64-ec2-013
bld-linux64-ec2-028
bld-linux64-ec2-037
bld-linux64-ec2-050
bld-linux64-ec2-090
bld-linux64-ec2-128
bld-linux64-ec2-167
bld-linux64-ec2-177
bld-linux64-ec2-187

Comment 7

[Gregory Szorc [:gps]]

(In reply to Taras Glek (:taras) from comment #3)
> No. This bug is about doing a no-brainer, straight-forward switch to save
> $$$. Using a bigger instance effectively(keeping 32 cores idle is big
> opportunity cost) will require some significant infra rework. 
> 
> Mike ran some windows builds on diff c3* nodes and a) their raw cpu perf
> doesn't seem to scale linearly b) cost goes up 
> 13:37 <taras> so $0.180 for c3.xlarge, $0.2 FOR c3.2xlarge and $0.32 for the
> other
> based on
> 13:34 <glandium> fwiw, a windows with mozmake on c3.xlarge takes ~35min, on
> c3.2xlarge, ~20min and on c3.4xlarge, ~16min
> 
> I'd like to work on moving to the fastest cpu nodes for building in the
> medium-term future, but for this year I'd simply like to simply make sure
> that we are paying what we should for infra we have now.

SGTM. FWIW, average CPU during builds has been creeping down due to unified sources work. It was around 94% on my 4+4 core machine. It's now 81-85%. We'll need to derecursify directory traversal for the libs and tools tiers to get us back up to where I'd like to be (100%).

Comment 8

[Mike Hommey [:glandium]]

(In reply to Gregory Szorc [:gps] (in southeast Asia until Dec 14) from comment #7)
> SGTM. FWIW, average CPU during builds has been creeping down due to unified
> sources work. It was around 94% on my 4+4 core machine. It's now 81-85%.
> We'll need to derecursify directory traversal for the libs and tools tiers
> to get us back up to where I'd like to be (100%).

It's more a matter of moving things that are not doomed by dependency issues out of libs to a parallelized tier than derecursifying. But this is offtopic.

Comment 9

[Mike Hommey [:glandium]]

Also, you're saying it like we've got slower, but non compile tiers have not become instantly slower. They just now represent a bigger percentage of the overall build time, since compile has dramatically improved. But again, offtopic.

Comment 10

[Rail Aliiev [:rail]]

Interesting, a lot of c3.xlarge instances have issues with puppet:

Dec  3 10:42:05 bld-linux64-ec2-128 kernel: puppet[1309] trap invalid opcode ip:329a814be0 sp:7fff2652c6e8 error:0 in ld-2.12.so[329a800000+20000]
Dec  3 10:42:07 bld-linux64-ec2-128 abrt[1367]: saved core dump of pid 1309 (/usr/bin/ruby) to /var/spool/abrt/ccpp-2013-12-03-10:42:05-1309.new/coredump (61222912 bytes)

Comment 11

[(dormant account)]

(In reply to Rail Aliiev [:rail] from comment #10)
> Interesting, a lot of c3.xlarge instances have issues with puppet:
> 
> Dec  3 10:42:05 bld-linux64-ec2-128 kernel: puppet[1309] trap invalid opcode
> ip:329a814be0 sp:7fff2652c6e8 error:0 in ld-2.12.so[329a800000+20000]
> Dec  3 10:42:07 bld-linux64-ec2-128 abrt[1367]: saved core dump of pid 1309
> (/usr/bin/ruby) to
> /var/spool/abrt/ccpp-2013-12-03-10:42:05-1309.new/coredump (61222912 bytes)

Mike, can you help figure out how to proceed?
Looks like this is an issue to do with cpu supporting avx(and exposing it via cpu flags), but having it not enabled on either xen or kernel level.

https://www.ruby-forum.com/topic/4074228

https://www.google.com/search?q=avx+xen+illegal+instruction&ie=utf-8&oe=utf-8&aq=t&rls=org.mozilla:en-US:unofficial&client=firefox-nightly&channel=sb

Comment 12

[Rail Aliiev [:rail]]

FTR, we use stock centos kernel (2.6.32-220.el6.x86_64) over pv-grub

Comment 13

[(dormant account)]

Rail, 
Can you try running http://vps.glek.net/avx_test.cpp(compile instructions on first line) on our c3 centos and on latest ubuntu c3?

Comment 14

[Rail Aliiev [:rail]]

"Illegal instruction" on both Centos (2.6.32-220.el6.x86_64, c3.xlarge) and Ubuntu (3.2.0-23-generic, m1.medium).

Comment 15

[(dormant account)]

(In reply to Rail Aliiev [:rail] from comment #14)
> "Illegal instruction" on both Centos (2.6.32-220.el6.x86_64, c3.xlarge) and
> Ubuntu (3.2.0-23-generic, m1.medium).

also fails on c3.xlarge with ubuntu 3.11.0-12-generic kernel...works on real avx machines

Comment 16

[(dormant account)]

(In reply to Taras Glek (:taras) from comment #15)
> (In reply to Rail Aliiev [:rail] from comment #14)
> > "Illegal instruction" on both Centos (2.6.32-220.el6.x86_64, c3.xlarge) and
> > Ubuntu (3.2.0-23-generic, m1.medium).
> 
> also fails on c3.xlarge with ubuntu 3.11.0-12-generic kernel...works on real
> avx machines

Figured it out. paravirtual xen kernels setups are broken. We should be using hvm, then it works fine.

Comment 17

[(dormant account)]

(In reply to Taras Glek (:taras) from comment #16)
> Figured it out. paravirtual xen kernels setups are broken. We should be
> using hvm, then it works fine.

Good news is that hvm is the future and apparently is a few percent more efficient for cpu and/or memory-intensive workloads

Comment 18

[(dormant account)]

Summary, we should use hvm(and a modern kernel) on ec2 nodes that support it. If we don't use hvm we will run into avx trouble and possibly lower throughput.

Comment 19

[Rail Aliiev [:rail]]

This sounds like we would need to create a separate AMI for HVM and "reimage" existing VMs.

I'm going to convert the instance mentioned above back to m3.xlarge now since they are useless as c3.xlarge.

Comment 20

[Rail Aliiev [:rail]]

(In reply to Rail Aliiev [:rail] from comment #19) 
> I'm going to convert the instance mentioned above back to m3.xlarge now
> since they are useless as c3.xlarge.

This is done now.

Comment 21

[Rail Aliiev [:rail]]

(In reply to Rail Aliiev [:rail] from comment #19)
> This sounds like we would need to create a separate AMI for HVM and
> "reimage" existing VMs.

This will require:
 * some changes to our AMI script  -- it should use real grub or even grub2 instead of plain pv-grub config
 * make sure that the stock kernel still works with this configuration

Back to the pool for volunteers until I have more time.

Comment 22

[Massimo Gervasini [:massimo]]

I am getting intermittent "Insufficient capacity" errors creating the AMI.

According to http://aws.typepad.com/aws/2013/12/c3-instance-update.html there is huge requests of this type of instances.

Comment 23

[Rail Aliiev [:rail]]

Attachment: HVM

This is mgerva's patch with some additions to handle instance storage. Comments incoming.

Comment 24

[Rail Aliiev [:rail]]

Comment on attachment 8362614 [details] [diff] [review]
HVM

Review of attachment 8362614 [details] [diff] [review]:
-----------------------------------------------------------------

::: aws/aws_create_ami.py
@@ +78,5 @@
> +             "/root/.ssh/authorized_keys" % user)
> +        sudo("sed -i -e '/PermitRootLogin/d' "
> +             "-e '$ a PermitRootLogin without-password' /etc/ssh/sshd_config")
> +        sudo("service sshd restart")
> +        sudo("sleep 20")

Some of the initial AMI doesn't allow root logins what adds some complexity to Fabric commands.

@@ +179,5 @@
>  
>      # Step 3: upload custom configuration files
>      run('chroot %s mkdir -p /boot/grub' % mount_point)
> +    with lcd(config_dir):
> +        for f in ('etc/rc.local', 'etc/fstab', 'etc/hosts',

This simplifies file copying and makes it distro independent. In the future it would be better to list the files programmatically.

@@ +227,5 @@
> +
> +    run("sed -i -e '/PermitRootLogin/d' -e '/UseDNS/d' "
> +        "-e '$ a PermitRootLogin without-password' "
> +        "-e '$ a UseDNS no' "
> +        "%s/etc/ssh/sshd_config" % mount_point)

A safer way to "patch" the config.

@@ +291,5 @@
>              ami.add_tag('Name', dated_target_name)
> +            if config["target"].get("tags"):
> +                for tag, value in config["target"]["tags"].items():
> +                    log.info("Tagging %s: %s", tag, value)
> +                    ami.add_tag(tag, value)

This is not used anywhere yet, but it'd be better to tag AMIs.

::: aws/instance_data/us-east-1.instance_data_dev.json
@@ -1,5 @@
>  {
>    "buildslave_password": "pass",
>    "buildbot_master": "10.12.48.14:1313",
> -  "puppet_masters": ["releng-puppet1.srv.releng.use1.mozilla.com", "releng-puppet2.srv.releng.use1.mozilla.com"],
> -  "home_tarball": "secrets/dev.tar.gz"

We don't need these anymore since we handle the secrets by puppet.

Comment 25

[Rail Aliiev [:rail]]

I put 20 c3 build slaves into production yesterday:

* bld-linux64-20{0..9} (us-east-1): https://secure.pub.build.mozilla.org/builddata/reports/slave_health/slave.html?name=bld-linux64-ec2-20

* bld-linux64-50{0..9} (us-west-2): https://secure.pub.build.mozilla.org/builddata/reports/slave_health/slave.html?name=bld-linux64-ec2-50

So far I don't see any issues with them. aws_stop_idle.py skips them for now.

Comment 26

[Rail Aliiev [:rail]]

try-linux64-ec2-100 and try-linux64-ec2-350 are try slaves with instance storage mounted as /builds (combined with LVM). They took their first jobs and look good so far.

Comment 27

[Rail Aliiev [:rail]]

All done here.

{bld,try}-linux64-ec2 instances are c3.xlarge now.

We still use m3.xlarge for spot instances, but nothing holds us from switching to c3 as well (except the fact that it's easier to get m3.xlarge on the spot market, afaik) -- just a one line per region in http://hg.mozilla.org/build/cloud-tools/file/f3dcef708047/aws/configs/watch_pending.cfg#l36