Status

Release Engineering
Platform Support
RESOLVED FIXED
4 years ago
3 years ago

People

(Reporter: (dormant account), Assigned: massimo)

Tracking

Firefox Tracking Flags

(Not tracked)

Details

Attachments

(1 attachment)

(Reporter)

Description

4 years ago
c3 have 1 more ecu, faster IO, faster networking, 50% less ram and costs 33% less. Overall we should be same speed or faster for less $$$. I asked glandium to doublecheck.

In general newer c* nodes seem to offer better value. We should be switching from m* to c* nodes.
I've set up a c3.xlarge and a m3.xlarge instances with the same OS (Ubuntu LTS, fwiw), and built current m-c on it.
Build times are identical on both when the mozilla-central clone and build objdir are on EBS (around 26:30, without ccache). But c3.xlarge also has internal storage (according to the description, they are SSD), and placing the m-c clone and the build objdir there instead of EBS makes the build one minute faster.

Comment 2

4 years ago
Should we not invest in the c3.2x, c3.4x, or even c3.8x? The price is linear with number of cores and the build system can utilize those cores. This would, of course, reduce turnaround times for checkins, resulting in faster try pushes, shorter tree closures, etc. The biggest issue I see is that a "build" job consists of not only the highly-concurrent building process, but a lot of single or low core tasks, such as packaging and |make check|. These CPU inefficient tasks would effectively waste money on a high core machine. Of course, we can look into offloading those tasks to other builders (e.g. bug 822394) or improving the jobs to do more of the post-build tasks in parallel.
(Reporter)

Comment 3

4 years ago
(In reply to Gregory Szorc [:gps] from comment #2)
> Should we not invest in the c3.2x, c3.4x, or even c3.8x? The price is linear
> with number of cores and the build system can utilize those cores. This
> would, of course, reduce turnaround times for checkins, resulting in faster
> try pushes, shorter tree closures, etc. The biggest issue I see is that a
> "build" job consists of not only the highly-concurrent building process, but
> a lot of single or low core tasks, such as packaging and |make check|. These
> CPU inefficient tasks would effectively waste money on a high core machine.
> Of course, we can look into offloading those tasks to other builders (e.g.
> bug 822394) or improving the jobs to do more of the post-build tasks in
> parallel.

No. This bug is about doing a no-brainer, straight-forward switch to save $$$. Using a bigger instance effectively(keeping 32 cores idle is big opportunity cost) will require some significant infra rework. 

Mike ran some windows builds on diff c3* nodes and a) their raw cpu perf doesn't seem to scale linearly b) cost goes up 
13:37 <taras> so $0.180 for c3.xlarge, $0.2 FOR c3.2xlarge and $0.32 for the other
based on
13:34 <glandium> fwiw, a windows with mozmake on c3.xlarge takes ~35min, on c3.2xlarge, ~20min and on c3.4xlarge, ~16min

I'd like to work on moving to the fastest cpu nodes for building in the medium-term future, but for this year I'd simply like to simply make sure that we are paying what we should for infra we have now.
I'm going to change the instance type for some builder right now and watch them this week. If we are satisfied with the results we can switch to c3.xlarge for all builders.
Assignee: nobody → rail
Try:

try-linux64-ec2-002
try-linux64-ec2-004
try-linux64-ec2-005
try-linux64-ec2-007
try-linux64-ec2-011
try-linux64-ec2-015
try-linux64-ec2-020
try-linux64-ec2-023
try-linux64-ec2-028
try-linux64-ec2-031
Production builders:

bld-linux64-ec2-003
bld-linux64-ec2-004
bld-linux64-ec2-012
bld-linux64-ec2-013
bld-linux64-ec2-028
bld-linux64-ec2-037
bld-linux64-ec2-050
bld-linux64-ec2-090
bld-linux64-ec2-128
bld-linux64-ec2-167
bld-linux64-ec2-177
bld-linux64-ec2-187

Comment 7

4 years ago
(In reply to Taras Glek (:taras) from comment #3)
> No. This bug is about doing a no-brainer, straight-forward switch to save
> $$$. Using a bigger instance effectively(keeping 32 cores idle is big
> opportunity cost) will require some significant infra rework. 
> 
> Mike ran some windows builds on diff c3* nodes and a) their raw cpu perf
> doesn't seem to scale linearly b) cost goes up 
> 13:37 <taras> so $0.180 for c3.xlarge, $0.2 FOR c3.2xlarge and $0.32 for the
> other
> based on
> 13:34 <glandium> fwiw, a windows with mozmake on c3.xlarge takes ~35min, on
> c3.2xlarge, ~20min and on c3.4xlarge, ~16min
> 
> I'd like to work on moving to the fastest cpu nodes for building in the
> medium-term future, but for this year I'd simply like to simply make sure
> that we are paying what we should for infra we have now.

SGTM. FWIW, average CPU during builds has been creeping down due to unified sources work. It was around 94% on my 4+4 core machine. It's now 81-85%. We'll need to derecursify directory traversal for the libs and tools tiers to get us back up to where I'd like to be (100%).
(In reply to Gregory Szorc [:gps] (in southeast Asia until Dec 14) from comment #7)
> SGTM. FWIW, average CPU during builds has been creeping down due to unified
> sources work. It was around 94% on my 4+4 core machine. It's now 81-85%.
> We'll need to derecursify directory traversal for the libs and tools tiers
> to get us back up to where I'd like to be (100%).

It's more a matter of moving things that are not doomed by dependency issues out of libs to a parallelized tier than derecursifying. But this is offtopic.
Also, you're saying it like we've got slower, but non compile tiers have not become instantly slower. They just now represent a bigger percentage of the overall build time, since compile has dramatically improved. But again, offtopic.
Interesting, a lot of c3.xlarge instances have issues with puppet:

Dec  3 10:42:05 bld-linux64-ec2-128 kernel: puppet[1309] trap invalid opcode ip:329a814be0 sp:7fff2652c6e8 error:0 in ld-2.12.so[329a800000+20000]
Dec  3 10:42:07 bld-linux64-ec2-128 abrt[1367]: saved core dump of pid 1309 (/usr/bin/ruby) to /var/spool/abrt/ccpp-2013-12-03-10:42:05-1309.new/coredump (61222912 bytes)
(Reporter)

Comment 11

4 years ago
(In reply to Rail Aliiev [:rail] from comment #10)
> Interesting, a lot of c3.xlarge instances have issues with puppet:
> 
> Dec  3 10:42:05 bld-linux64-ec2-128 kernel: puppet[1309] trap invalid opcode
> ip:329a814be0 sp:7fff2652c6e8 error:0 in ld-2.12.so[329a800000+20000]
> Dec  3 10:42:07 bld-linux64-ec2-128 abrt[1367]: saved core dump of pid 1309
> (/usr/bin/ruby) to
> /var/spool/abrt/ccpp-2013-12-03-10:42:05-1309.new/coredump (61222912 bytes)

Mike, can you help figure out how to proceed?
Looks like this is an issue to do with cpu supporting avx(and exposing it via cpu flags), but having it not enabled on either xen or kernel level.

https://www.ruby-forum.com/topic/4074228

https://www.google.com/search?q=avx+xen+illegal+instruction&ie=utf-8&oe=utf-8&aq=t&rls=org.mozilla:en-US:unofficial&client=firefox-nightly&channel=sb
FTR, we use stock centos kernel (2.6.32-220.el6.x86_64) over pv-grub
(Reporter)

Comment 13

4 years ago
Rail, 
Can you try running http://vps.glek.net/avx_test.cpp(compile instructions on first line) on our c3 centos and on latest ubuntu c3?
"Illegal instruction" on both Centos (2.6.32-220.el6.x86_64, c3.xlarge) and Ubuntu (3.2.0-23-generic, m1.medium).
(Reporter)

Comment 15

4 years ago
(In reply to Rail Aliiev [:rail] from comment #14)
> "Illegal instruction" on both Centos (2.6.32-220.el6.x86_64, c3.xlarge) and
> Ubuntu (3.2.0-23-generic, m1.medium).

also fails on c3.xlarge with ubuntu 3.11.0-12-generic kernel...works on real avx machines
(Reporter)

Comment 16

4 years ago
(In reply to Taras Glek (:taras) from comment #15)
> (In reply to Rail Aliiev [:rail] from comment #14)
> > "Illegal instruction" on both Centos (2.6.32-220.el6.x86_64, c3.xlarge) and
> > Ubuntu (3.2.0-23-generic, m1.medium).
> 
> also fails on c3.xlarge with ubuntu 3.11.0-12-generic kernel...works on real
> avx machines

Figured it out. paravirtual xen kernels setups are broken. We should be using hvm, then it works fine.
(Reporter)

Comment 17

4 years ago
(In reply to Taras Glek (:taras) from comment #16)
> Figured it out. paravirtual xen kernels setups are broken. We should be
> using hvm, then it works fine.

Good news is that hvm is the future and apparently is a few percent more efficient for cpu and/or memory-intensive workloads
(Reporter)

Comment 18

4 years ago
Summary, we should use hvm(and a modern kernel) on ec2 nodes that support it. If we don't use hvm we will run into avx trouble and possibly lower throughput.
This sounds like we would need to create a separate AMI for HVM and "reimage" existing VMs.

I'm going to convert the instance mentioned above back to m3.xlarge now since they are useless as c3.xlarge.
(In reply to Rail Aliiev [:rail] from comment #19) 
> I'm going to convert the instance mentioned above back to m3.xlarge now
> since they are useless as c3.xlarge.

This is done now.
(Reporter)

Updated

4 years ago
Blocks: 945981
(In reply to Rail Aliiev [:rail] from comment #19)
> This sounds like we would need to create a separate AMI for HVM and
> "reimage" existing VMs.

This will require:
 * some changes to our AMI script  -- it should use real grub or even grub2 instead of plain pv-grub config
 * make sure that the stock kernel still works with this configuration

Back to the pool for volunteers until I have more time.
Assignee: rail → nobody
Component: Other → Platform Support
QA Contact: joduinn → coop
(Assignee)

Updated

4 years ago
Assignee: nobody → mgervasini
(Assignee)

Comment 22

4 years ago
I am getting intermittent "Insufficient capacity" errors creating the AMI.

According to http://aws.typepad.com/aws/2013/12/c3-instance-update.html there is huge requests of this type of instances.
Created attachment 8362614 [details] [diff] [review]
HVM

This is mgerva's patch with some additions to handle instance storage. Comments incoming.
Attachment #8362614 - Flags: review?(mgervasini)
Attachment #8362614 - Flags: review?(catlee)
Comment on attachment 8362614 [details] [diff] [review]
HVM

Review of attachment 8362614 [details] [diff] [review]:
-----------------------------------------------------------------

::: aws/aws_create_ami.py
@@ +78,5 @@
> +             "/root/.ssh/authorized_keys" % user)
> +        sudo("sed -i -e '/PermitRootLogin/d' "
> +             "-e '$ a PermitRootLogin without-password' /etc/ssh/sshd_config")
> +        sudo("service sshd restart")
> +        sudo("sleep 20")

Some of the initial AMI doesn't allow root logins what adds some complexity to Fabric commands.

@@ +179,5 @@
>  
>      # Step 3: upload custom configuration files
>      run('chroot %s mkdir -p /boot/grub' % mount_point)
> +    with lcd(config_dir):
> +        for f in ('etc/rc.local', 'etc/fstab', 'etc/hosts',

This simplifies file copying and makes it distro independent. In the future it would be better to list the files programmatically.

@@ +227,5 @@
> +
> +    run("sed -i -e '/PermitRootLogin/d' -e '/UseDNS/d' "
> +        "-e '$ a PermitRootLogin without-password' "
> +        "-e '$ a UseDNS no' "
> +        "%s/etc/ssh/sshd_config" % mount_point)

A safer way to "patch" the config.

@@ +291,5 @@
>              ami.add_tag('Name', dated_target_name)
> +            if config["target"].get("tags"):
> +                for tag, value in config["target"]["tags"].items():
> +                    log.info("Tagging %s: %s", tag, value)
> +                    ami.add_tag(tag, value)

This is not used anywhere yet, but it'd be better to tag AMIs.

::: aws/instance_data/us-east-1.instance_data_dev.json
@@ -1,5 @@
>  {
>    "buildslave_password": "pass",
>    "buildbot_master": "10.12.48.14:1313",
> -  "puppet_masters": ["releng-puppet1.srv.releng.use1.mozilla.com", "releng-puppet2.srv.releng.use1.mozilla.com"],
> -  "home_tarball": "secrets/dev.tar.gz"

We don't need these anymore since we handle the secrets by puppet.
I put 20 c3 build slaves into production yesterday:

* bld-linux64-20{0..9} (us-east-1): https://secure.pub.build.mozilla.org/builddata/reports/slave_health/slave.html?name=bld-linux64-ec2-20

* bld-linux64-50{0..9} (us-west-2): https://secure.pub.build.mozilla.org/builddata/reports/slave_health/slave.html?name=bld-linux64-ec2-50

So far I don't see any issues with them. aws_stop_idle.py skips them for now.
Depends on: 962099
Depends on: 962383
try-linux64-ec2-100 and try-linux64-ec2-350 are try slaves with instance storage mounted as /builds (combined with LVM). They took their first jobs and look good so far.

Updated

4 years ago
Attachment #8362614 - Flags: review?(catlee) → review+
(Assignee)

Updated

4 years ago
Attachment #8362614 - Flags: review?(mgervasini) → review+

Updated

3 years ago
Depends on: 964880
All done here.

{bld,try}-linux64-ec2 instances are c3.xlarge now.

We still use m3.xlarge for spot instances, but nothing holds us from switching to c3 as well (except the fact that it's easier to get m3.xlarge on the spot market, afaik) -- just a one line per region in http://hg.mozilla.org/build/cloud-tools/file/f3dcef708047/aws/configs/watch_pending.cfg#l36
Status: NEW → RESOLVED
Last Resolved: 3 years ago
Resolution: --- → FIXED
OS: Windows 8 → Linux
You need to log in before you can comment on or make changes to this bug.