Closed Bug 987303 Opened 6 years ago Closed 6 years ago

evaluate mac cloud options


(Infrastructure & Operations Graveyard :: CIDuty, task)

Not set


(Not tracked)



(Reporter: kmoir, Assigned: kmoir)


Details and perhaps others.  Evaluate trial vms and see if they would work, issues etc.
Assignee: nobody → kmoir
So these are the criteria I had in mind for looking at minis
1) Be real racked minis not virtual
2) Full administrative access to the machines so we can puppetize them to our specifications
3) Support to install the mac os versions we need to test, not just support for the latest version
4) Network access similar to Amazon VPC so we can connect them to to our build farm via appropriate ports etc
5) ability to scale to our needs ... sufficient capacity to replace at least mac pool as a trial
6) cost considerations

Not sure if there are more criteria to evaluate.  I plan to look at the technical specifications first because this will weed out a lot of companies and then evaluate on price.

So I looked at these options
I wrote them they can do puppet and network access to vpc, have capacity.  Are willing to spin up a few test servers for us to try out.
-based in Zurich, looks like these are virtual servers not just racked minis so not relevant for running tests

I was looking at their page and then saw they they advertize their datacenter with a woman wearing hot pants in front of their mac mini racks.  So I closed that tab.  Maybe I'll go back and look at them later but now I really don't feel like it. Machines are awesome enough on their own, you don't need sexist pictures to sell them.
Hmm, seems like pretty small scale.  Not really enterprise level hosting.
They are mostly a colo, although you can rent them and after 12 months you own them.  Not sure if this is right for us since we are trying to get out of the owning mac minis business.
Not sure if you can puppetize their servers, will write and find out.

More information here but it is a bit out of date

Will update the bug as I find out more information.
Depends on: 997943
Depends on: 997965
Depends on: 1003171
So to update current status on this bug

I gave a presentation last week at the releng work week on the current status of my investigation

I have run M1 opt tests on a mac provided by a vendor and they ran in similar time to that on our slaves.  That being said the vendor slaves are better hardware.  I'll run tests on the vendor other slaves too and the larger suite of tests.

As a side note, the criteria in the presentation for the vendors is as follows
-Be real macs not virtual (to preserve talos numbers)
-Full admin access
-Support for multiple images
-Sufficient capacity to scale to our needs
-network configuration similar to AWS VPC
-cost considerations

more criteria that were raise by attendees at the work week
-people on site to reboot machines if they can't be rebooted remotely
-machine to machine connect for reboots. sshd alone isn't sufficient for reboots
-automated reimaging
Further updates
-yes both vendors have people on site to reboot machines if they can't be rebooted remotely
-yes to machine machine reboot from both vendors
-automated imaging is available
-Also one of the vendor raised this issue related to this  Basically as soon as Apple releases new hardware (perhaps at the next WWDC) you can't run older osx versions on new hardware.  I.e. 10.9.0 might not run on new hardware, but 10.9.x will.  They release new builds as soon as the new hardware is released with new drivers etc.
Regarding comment #3, I should have mentioned that the the context with the new hardware is that obviously we have a large number of machines in our pool and the vendors don't have the amount of excess capacity around for us, they would have to buy new hardware.  Thus point about the hardware is that we may need to run on 10.9.x on newer hardware instead of 10.9.0.
Depends on: 1017207
So WWDC was yesterday.  No new hardware was announced.  OSX 10.10 was announced but will be in beta until the fall.  

The drive to get the 10.9 tests greened up is slow going.  (There are developers/ateam people assigned to it but they are busy with other priorities)  Last week in our 1x1 coop suggested that I look at running builds on 10.9.  If this worked out, we could move 10.9 builds to the third party colo and reallocate our existing macosx build machines to run as 10.8 test machines.  This would make the 10.8 test pool bigger and presumably reduce the wait times.  

I've asked relops for a 10.9 image in bug 1017207 and once this is ready it will be installed at the vendors and I'll work on getting the necessary changes configured so that I can test running builds there.
Kim, great work. Looking forward to see us using this for 10.9
Depends on: 1021892
So I have evaluated several vendors from a technical and cost perspective.  I have opened bugs with Finance and IT as required.  Now we just have to decide if the cost is worth it given our current costs.
(In reply to Kim Moir [:kmoir] from comment #7)
> So I have evaluated several vendors from a technical and cost perspective. 
> I have opened bugs with Finance and IT as required.  Now we just have to
> decide if the cost is worth it given our current costs.

Where do you have costs posted? Can you cc me on finance bug?
Flags: needinfo?(kmoir)

I have some costs from a vendor but I'm collecting internal costs to make a comparison to see if it's value for money. I'll send you an email with details.  Costs for macs in a colo are very very expensive compared to our AWS costs per node.
Flags: needinfo?(kmoir)
We have delayed this project for a quarter until so we can complete other priorities.
This can be closed for now since it's not on a priorities list for releng for this quarter.
Closed: 6 years ago
Resolution: --- → WONTFIX
Component: Platform Support → Buildduty
Product: Release Engineering → Infrastructure & Operations
Product: Infrastructure & Operations → Infrastructure & Operations Graveyard
You need to log in before you can comment on or make changes to this bug.