688632 - evaluate new mini hardware for rev5 builders

Status: RESOLVED FIXED

(Release Engineering :: General, defect, P2)

Description

[John O'Duinn [:joduinn] (please use "needinfo?" flag)]

Attachment: build.sh

Before we go ahead and order rev5 minis for OSX builders, we'd like to do a brief cost/performance tradeoff of some interesting configurations.
 
The mini in bug#675843 was imaged with 10.6.8 and used to build full-clobber nightly builds in a loop for a few weeks now, all based on pulling tip of mozilla-central. For later reference, the quick shell script, and the mozconfig used is attached, and below are some build times:
...
3h29m
3h29m
3h29m
3h29m
3h33m
3h29m
3h29m

Tomorrow, jhford is getting a loan of three other candidate rev5 minis from zandr. All have different CPU/RAM/disk configurations - more details once we have the machines in hand. We'll try running clobber nightly builds on those and see which works best for us.

Comment 1

[John O'Duinn [:joduinn] (please use "needinfo?" flag)]

Attachment: mozconfig used for these clobber builds

Comment 2

[John O'Duinn [:joduinn] (please use "needinfo?" flag)]

Attachment: time for builds, in seconds-since-epoch

Comment 3

[Zandr Milewski [:zandr]]

(In reply to John O'Duinn [:joduinn] from comment #0)

> The mini in bug#675843 was imaged with 10.6.8 and used to build full-clobber

Which is not supported, FWIW. That machine shipped with 10.7, and 10.6.8 is not supported on that platform, likely to have issues with SSDs, etc. When evaluating that configuration relative to others, you should reinstall 10.7 and repeat the tests.

> All have different CPU/RAM/disk configurations - more details once we
> have the machines in hand. We'll try running clobber nightly builds on those
> and see which works best for us.

This was in in email to joduinn/bmoss:

05441 2.0GHz Quad Core/256GB SSD+750GB 7200rpm/8GB/Intel HD3000 (Server)
05442 2.7GHz Dual Core/256GB SSD+750GB 7200rpm/8GB/Radeon HD6630M
05443 2.3GHz Dual Core/500GB 5400rpm          /8GB/Intel HD3000

The low-end configuration is about half the money, so if it's more than half the performance, it's worth doing some math.

Also, ccache will be a big win here, though it complicates that looping test.

Comment 4

[John Ford [:jhford] CET/CEST Berlin Time]

I have numbers from the Rev2 machine, using the same script and the same mozconfig file.

2:48
2:48
2:48
2:48
2:48
2:48

(yes, they all were within a couple seconds of 2:48).

This script only tests the "make -f client.mk build" time, which is a (very large) subset of all the things that take large amounts of time on these machines.

Once I get the 5 minis building, with a more complete script, I will report on performance numbers with 10.6.  Regarding comment 3, my intention was to do the tests using 10.7.  I am starting with 10.6 because a) I want to do an apples to apples comparison between hardware and b) at this point, 10.6 is a requirement.  Once I have numbers for building on 10.6, I will reimage the machines with 10.7 and do some speculative tests.  I say speculative because we don't currently know that we can use 10.7 for builds.  I will keep posting to this bug with information as I get it.

As for ccache, I don't think it is going to complicate things drastically.  Because we are able to explicitly clear the cache, we can build this into our timing script.

Also, I have two machines that are Core i7 2.7GHz spec.  I am going to image one with the spinning disc and one with the ssd to figure out if the SSDs are worth their price for our workload.

Comment 5

[Dustin J. Mitchell [:dustin] (he/him)]

Any update here?  We need to get moving on this r5 project..

Comment 6

[John Ford [:jhford] CET/CEST Berlin Time]

(In reply to Dustin J. Mitchell [:dustin] from comment #5)
> Any update here?  We need to get moving on this r5 project..

No.  I have been focused on pgo, hp-linu, 10.7 and 10.6 rev3 machines.

Comment 7

[John Ford [:jhford] CET/CEST Berlin Time]

Also, we are blocked on build system work.

Comment 8

[Chris Cooper [:coop] (he/him)]

(In reply to John Ford [:jhford] from comment #7)
> Also, we are blocked on build system work.

Which bug is this? Bug 674655?

Is there any value is pushing forward to get these machines building with the SDKs we already know about (10.7/10.6)?

Comment 9

[John Ford [:jhford] CET/CEST Berlin Time]

I still don't know what bug is tracking the build system issues, but the work is to be able to build 10.5/10.6/10.7 on 10.7 os

Running 10.6 on the rev5 hardware has major issues, like taking an hour longer than rev2 hardware with the same software.  There were also issues with display drivers that seem to be related to the intel integrated display adapter (models with intel+ati didn't exhibit the issues).

if we want to know "which can build firefox faster" with an unsupported configuration (10.7 + xcode 4.?), we could continue these tests.  I think that is a generally useful number, especially if we have to make a purchasing decision before we the final supported config.

Comment 10

[Amy Rich [:arr] [:arich]]

Coop/John pointed out that you may be blocked from getting this work done on IT issues, but I don't recall seeing any bugs open for us.  If you are still blocked, can you please point me at an existing bug or open one so we can get the issue solved?
If you're not blocked, can you please confirm that?  Thanks!

Comment 11

[John Ford [:jhford] CET/CEST Berlin Time]

We can get a general idea of which hardware spec is the best for building firefox on 10.7.  Because we are blocked on build system work to get a fully supported toolchain, we can only get approximate numbers using Xcode 4 and gcc.

I only have one rev5 machine at my desk.  There are a couple of variables that I'd like to account for:

1) apple ssd vs apple hd
2) top-of-line quad core vs top-of-line dual core

I don't personally want to bother with slower machines than the top of the line dual or quad core, but we could also look at:

3) cheap dual core vs top-of-line dual core

As I already have a top-of-line dual core with SSD, I think I'd need to have the following machines to test for all three variables:

dual-2.7 i7 with hard disc -- compare to dual i7 with ssd for ssd vs hd
quad-2.0 i7 with ssd -- compare to dual i7 with ssd for quad vs dual core
dual-2.3 i5 with hard disc -- compare to dual i7 with hard disc for cheap vs fast dual core

I am blocked on having the above hardware available to me.  Should I file a bug to have those machines made available or is this bug a good place for that?

Comment 12

[Zandr Milewski [:zandr]]

I would recommend ordering the dual-2.7 and quad-2.0 with both ssd and hard disks. There are two drive bays in the new machines, and I would recommend testing each configuration directly, rather than trying to control for changing two variables at once.

Thus:
dual-2.7 i7 with 7200rpm HD and SSD
quad-2.0 i7 with 7200rpm HD and SSD
dual-2.3 i5 with 5400rpm HD

Please file a Server Operations: Desktop bug to get these machines ordered, trying to track that in here is a recipe for failure.

Comment 13

[Dustin J. Mitchell [:dustin] (he/him)]

John - please link from here to the desktop bug for these units.  This is quickly going to become time-critical, so please do so today.  As that the machines be delivered directly to you in sfo

Comment 14

[John Ford [:jhford] CET/CEST Berlin Time]

(In reply to Dustin J. Mitchell [:dustin] from comment #13)
> John - please link from here to the desktop bug for these units.  This is
> quickly going to become time-critical, so please do so today.  As that the
> machines be delivered directly to you in sfo

Done, bug 701883

Comment 15

[Zandr Milewski [:zandr]]

The benchmark script just grabs tip, could we pick a tag so we can compare results?

Comment 16

[John Ford [:jhford] CET/CEST Berlin Time]

(In reply to Zandr Milewski [:zandr] from comment #15)
> The benchmark script just grabs tip, could we pick a tag so we can compare
> results?

Yep, that's the plan.

Comment 17

[Dustin J. Mitchell [:dustin] (he/him)]

John: please pick a tag or rev and post it here.

Comment 18

[John Ford [:jhford] CET/CEST Berlin Time]

(In reply to Dustin J. Mitchell [:dustin] from comment #17)
> John: please pick a tag or rev and post it here.

bugmail for this got lost in holiday bugspam rush.

I am working with tools out of https://github.com/jhford/mozilla-build-benchmarks, which includes some of my raw data.  I am working on analysis right now, as well as gathering more data around possible clang usage and -j make flags.

Comment 19

[Zandr Milewski [:zandr]]

(In reply to John Ford [:jhford] from comment #18)
> I am working with tools out of
> https://github.com/jhford/mozilla-build-benchmarks, 

A bit of source spelunking leads me to

REV=b87861e50640

in bench.sh Is that what we want to use going forward?

Comment 20

[John Ford [:jhford] CET/CEST Berlin Time]

(In reply to Zandr Milewski [:zandr] from comment #19)
> (In reply to John Ford [:jhford] from comment #18)
> > I am working with tools out of
> > https://github.com/jhford/mozilla-build-benchmarks, 
> 
> A bit of source spelunking leads me to
> 
> REV=b87861e50640

Yep, that's what I am using

> in bench.sh Is that what we want to use going forward?

Yes, that's the revision I plan on using for the whole set of benchmarks

Comment 21

[Dustin J. Mitchell [:dustin] (he/him)]

John, do you have a guess as to when you'll have a unit selected?  I'm trying to sketch out schedules, so provisional is fine.

Comment 22

[Chris Cooper [:coop] (he/him)]

John was traveling yesterday, so I'll sum up what he's told me so far:

* i7 are (understandably) markedly faster than i5s in the timings he's done so far
* current decision is between 2.7GHz dual-core i7 vs. 2.0GHz quad-core i7 (timings in progress)
* jhford also plans to run tests with various job settings (-jX) so we can be building optimally from the get-go

Comment 23

[Dustin J. Mitchell [:dustin] (he/him)]

(In reply to Chris Cooper [:coop] from comment #22)
> John was traveling yesterday, so I'll sum up what he's told me so far:
> 
> * i7 are (understandably) markedly faster than i5s in the timings he's done
> so far
> * current decision is between 2.7GHz dual-core i7 vs. 2.0GHz quad-core i7
> (timings in progress)

So it sounds like this will determine the spec we want, which just leaves a decision on quantity, and that will happen soon (like, early next week?)

> * jhford also plans to run tests with various job settings (-jX) so we can
> be building optimally from the get-go

This can continue after we've placed the order with Apple, right?

We'll also need to get the image creation rolling at this point - at least, before we have the delivery from Apple (which took a month last time -- too long to wait to start the image).  We can work out how to get the eval minis on the build network when we start that process.

Comment 24

[John Ford [:jhford] CET/CEST Berlin Time]

(In reply to Dustin J. Mitchell [:dustin] from comment #23)
> > * jhford also plans to run tests with various job settings (-jX) so we can
> > be building optimally from the get-go
> 
> This can continue after we've placed the order with Apple, right?

Unless we can modify an order for dual-core to quad-core, or vice-versa, I don't think we can say yet.  Also, I think it would also be prudent to wait on verifying that the 10.5-compatible builds generated on 10.7 are acceptable.  If we can't build 10.5 on 10.7 and supporting 10.5 remains a requirement, I don't know that we should proceed with this order.

Doing the testing of -jX will give us information that will allow us to pick between dual and quad core specs.  I have the data for making the HD vs SSD choice, at least from a performance perspective.

Comment 25

[Dustin J. Mitchell [:dustin] (he/him)]

We don't have time to serialize this hardware deployment with the development work at this point.  At any rate, we'll need to build on Lion at some point, and we'll need hardware to do that.  The work in this bug is *not* blocked on development, and is a high priority.

We definitely can't modify the order, and we'll absolutely wait until you have a specific model and quantity.  Coop's comments led me to think you were experimenting with build parameters separately from selecting hardware, so thanks for clarifying.  If you have an ETA for a final determination, it would help my planning.

Comment 26

[John Ford [:jhford] CET/CEST Berlin Time]

(In reply to Dustin J. Mitchell [:dustin] from comment #25)
> We definitely can't modify the order, and we'll absolutely wait until you
> have a specific model and quantity.  Coop's comments led me to think you
> were experimenting with build parameters separately from selecting hardware,
> so thanks for clarifying.  If you have an ETA for a final determination, it
> would help my planning.

I have a script that does the builds and the analysis, so the time spent is mainly in running the builds.  I plan on spending around 8-10 hours gathering data for each -j setting, so a few days will be needed to gather the data.  As for my evaluation, I will have numbers and my recommendation for hardware spec by Wednesday Jan 11 or sooner.  I don't know how long it will take before we get purchasing sign off on a spec of mini after that.

Comment 27

[Zandr Milewski [:zandr]]

2x2.7 vs. 4x2.0 is only one parameter. We also need to come to a conclusion on SSD vs. HDD (or both)

Please continue to collect data on the i5 machines as well, they're slower, but they're substantially cheaper, so !/$ may actually be higher. (Which, granted, is a complex calculation when you start talking about additional machines)

Comment 28

[John Ford [:jhford] CET/CEST Berlin Time]

(In reply to Zandr Milewski [:zandr] from comment #27)
> 2x2.7 vs. 4x2.0 is only one parameter. We also need to come to a conclusion
> on SSD vs. HDD (or both)

Yes, i had mentioned that I have data to do hd vs ssd comparisons.  I am interested in the difference between performance between ssd and hdd when the -j setting is increased as well.  I only have dual core machines with the OS and data on both ssd and hd.  Given that, I have found that while using -j8 overall slows down the dual core i7 build, the performance difference between ssd and hd increases.  This suggests that a quad-core with ssd would be significantly faster than a properly loaded quad-core with hd.

> Please continue to collect data on the i5 machines as well, they're slower,
> but they're substantially cheaper, so !/$ may actually be higher. (Which,
> granted, is a complex calculation when you start talking about additional
> machines)

Yes, I am collecting data on the i5 machine.  So far, my data is suggesting that using -j4 is better than -j8 on all dual core machines, and that the i5 is significantly worse than the other machines in every way.  Since I don't have pricing information, the only thing I can analyze is raw performance.  It will be up to the person who signs off on the purchase to make the call on price/performance.

Comment 29

[Zandr Milewski [:zandr]]

(In reply to John Ford [:jhford] from comment #28)

> This suggests that a quad-core
> with ssd would be significantly faster than a properly loaded quad-core with
> hd.

The first time around, I got significantly better performance putting OS and source on SSD, and the objdir on spinning metal. Are you doing any testing using *both* SSD an HD?

Comment 30

[John Ford [:jhford] CET/CEST Berlin Time]

(In reply to Zandr Milewski [:zandr] from comment #29)
> (In reply to John Ford [:jhford] from comment #28)
> 
> > This suggests that a quad-core
> > with ssd would be significantly faster than a properly loaded quad-core with
> > hd.
> 
> The first time around, I got significantly better performance putting OS and
> source on SSD, and the objdir on spinning metal. Are you doing any testing
> using *both* SSD an HD?

I haven't tested that.

Comment 31

[John Ford [:jhford] CET/CEST Berlin Time]

I have code and data in my github as well as a more indepth data on my blog.

The fastest is Quad-i7+SSD followed by Dual-i7+SSD, Dual-i7+HD then Dual-i5+HD


blog: http://blog.johnford.info/new-mac-builders-ssds-j-settings/
code: https://github.com/jhford/mozilla-build-benchmarks

Comment 32

[Zandr Milewski [:zandr]]

(In reply to John Ford [:jhford] from comment #31)
> I have code and data in my github as well as a more indepth data on my blog.

Your blog post raises a ton of questions in my mind. Would you like that discussion here or in comments on your blog?