987303 - evaluate mac cloud options

Status: RESOLVED WONTFIX

(Infrastructure & Operations Graveyard :: CIDuty, task)

Description

[Kim Moir [:kmoir] ET]

macincloud.com and perhaps others.  Evaluate trial vms and see if they would work, issues etc.

Comment 1

[Kim Moir [:kmoir] ET]

So these are the criteria I had in mind for looking at minis
1) Be real racked minis not virtual
2) Full administrative access to the machines so we can puppetize them to our specifications
3) Support to install the mac os versions we need to test, not just support for the latest version
4) Network access similar to Amazon VPC so we can connect them to to our build farm via appropriate ports etc
5) ability to scale to our needs ... sufficient capacity to replace at least mac pool as a trial
6) cost considerations

Not sure if there are more criteria to evaluate.  I plan to look at the technical specifications first because this will weed out a lot of companies and then evaluate on price.

So I looked at these options
http://www.macincloud.com/features/tools/tools
I wrote them they can do puppet and network access to vpc, have capacity.  Are willing to spin up a few test servers for us to try out.

http://xcloud.me/pricing-signup/
-based in Zurich, looks like these are virtual servers not just racked minis so not relevant for running tests

Macstadium
https://macstadium.com/datacenter
I was looking at their page and then saw they they advertize their datacenter with a woman wearing hot pants in front of their mac mini racks.  So I closed that tab.  Maybe I'll go back and look at them later but now I really don't feel like it. Machines are awesome enough on their own, you don't need sexist pictures to sell them.

http://www.xcodeclub.com/
Hmm, seems like pretty small scale.  Not really enterprise level hosting.

http://www.macminivault.com/
They are mostly a colo, although you can rent them and after 12 months you own them.  Not sure if this is right for us since we are trying to get out of the owning mac minis business.

http://virtualmacosx.com/index.php/server-plans
Not sure if you can puppetize their servers, will write and find out.

More information here but it is a bit out of date
http://stackoverflow.com/questions/7308039/do-on-demand-mac-os-x-cloud-services-exist-comparable-to-amazons-ec2-on-demand

Will update the bug as I find out more information.

Comment 2

[Kim Moir [:kmoir] ET]

So to update current status on this bug

I gave a presentation last week at the releng work week on the current status of my investigation

https://wiki.mozilla.org/ReleaseEngineering/Workweeks/Portland-April2014#Presentations.2C_shareables_.26_recordings

I have run M1 opt tests on a mac provided by a vendor and they ran in similar time to that on our slaves.  That being said the vendor slaves are better hardware.  I'll run tests on the vendor other slaves too and the larger suite of tests.

As a side note, the criteria in the presentation for the vendors is as follows
-Be real macs not virtual (to preserve talos numbers)
-Full admin access
-Support for multiple images
-Sufficient capacity to scale to our needs
-network configuration similar to AWS VPC
-cost considerations

more criteria that were raise by attendees at the work week
-people on site to reboot machines if they can't be rebooted remotely
-machine to machine connect for reboots. sshd alone isn't sufficient for reboots
-automated reimaging

Comment 3

[Kim Moir [:kmoir] ET]

Further updates
-yes both vendors have people on site to reboot machines if they can't be rebooted remotely
-yes to machine machine reboot from both vendors
-automated imaging is available
-Also one of the vendor raised this issue related to this http://support.apple.com/kb/HT1159.  Basically as soon as Apple releases new hardware (perhaps at the next WWDC) you can't run older osx versions on new hardware.  I.e. 10.9.0 might not run on new hardware, but 10.9.x will.  They release new builds as soon as the new hardware is released with new drivers etc.

Comment 4

[Kim Moir [:kmoir] ET]

Regarding comment #3, I should have mentioned that the the context with the new hardware is that obviously we have a large number of machines in our pool and the vendors don't have the amount of excess capacity around for us, they would have to buy new hardware.  Thus point about the hardware is that we may need to run on 10.9.x on newer hardware instead of 10.9.0.

Comment 5

[Kim Moir [:kmoir] ET]

So WWDC was yesterday.  No new hardware was announced.  OSX 10.10 was announced but will be in beta until the fall.  

The drive to get the 10.9 tests greened up is slow going.  (There are developers/ateam people assigned to it but they are busy with other priorities)  Last week in our 1x1 coop suggested that I look at running builds on 10.9.  If this worked out, we could move 10.9 builds to the third party colo and reallocate our existing macosx build machines to run as 10.8 test machines.  This would make the 10.8 test pool bigger and presumably reduce the wait times.  

I've asked relops for a 10.9 image in bug 1017207 and once this is ready it will be installed at the vendors and I'll work on getting the necessary changes configured so that I can test running builds there.

Comment 6

[(dormant account)]

Kim, great work. Looking forward to see us using this for 10.9

Comment 7

[Kim Moir [:kmoir] ET]

So I have evaluated several vendors from a technical and cost perspective.  I have opened bugs with Finance and IT as required.  Now we just have to decide if the cost is worth it given our current costs.

Comment 8

[(dormant account)]

(In reply to Kim Moir [:kmoir] from comment #7)
> So I have evaluated several vendors from a technical and cost perspective. 
> I have opened bugs with Finance and IT as required.  Now we just have to
> decide if the cost is worth it given our current costs.

Where do you have costs posted? Can you cc me on finance bug?

Comment 9

[Kim Moir [:kmoir] ET]

:taras

I have some costs from a vendor but I'm collecting internal costs to make a comparison to see if it's value for money. I'll send you an email with details.  Costs for macs in a colo are very very expensive compared to our AWS costs per node.

Comment 10

[Kim Moir [:kmoir] ET]

We have delayed this project for a quarter until so we can complete other priorities.

Comment 11

[Kim Moir [:kmoir] ET]

This can be closed for now since it's not on a priorities list for releng for this quarter.