A flock of new machines for SeaMonkey

VERIFIED FIXED

Status

mozilla.org
Community Giving
VERIFIED FIXED
10 years ago
9 years ago

People

(Reporter: Robert Kaiser, Assigned: mrz)

Tracking

Details

(Reporter)

Description

10 years ago
In the light of branching coming up for Mozilla 1.9.1 soon, and for SeaMonkey probably some time between now and some point in Q1 of 2009, as well as requests for a debug build/test box and x86_64 Linux builds, and even trunk deprecation of Tiger possibly coming up soon, I'd like to request a number of machines for SeaMonkey.

In the ideal case, my vision is to get the following machines:
- 2 VMs with the Linux centos5 reference image,
  going into the current build/test pools for now,
  to have the resources for both trunk/branch once we branch SeaMonkey 2
- 2 VMs with the Win2k3 reference image,
  going into the current build/test pools for now,
  to have the resources for both trunk/branch once we branch SeaMonkey 2
- 1 VM with a Linux x86_64 system (do we have a refimage there?),
  for doing 64bit builds
- 1 VMs with the Linux centos5 reference image,
  for building debug, see bug 452147
- 2 Mac machines (min or xserve, whatever IT is more comfortable with),
  those should be Intel-based machines with Leopard installed,
  going into the current build/test pools for now,
  to have the resources for both trunk/branch once we branch SeaMonkey 2

This results in 6 VMs and 2 Macs, which is more machines than we even have right now, actually. I suspect this may raise some eyebrows, given we're a small project for a pretty niche product, but then, we never had a branch to maintain next to trunk apart from what I've been running on private machines. If money is a concern, maybe needing an extra ESX server to be added, or for the Macs, we have some amount of SeaMonkey-directed donations waiting at MoFo to be used, which we could possibly throw into this - but those possible concerns also why I'm filing this with enough time before our actual branching, so everything can be planned without too much pressure.
(Assignee)

Updated

10 years ago
Assignee: server-ops → sethb
(Assignee)

Comment 1

10 years ago
Passing over to community giving.
Component: Server Operations → Community Giving
QA Contact: mrz → community-giving
(Assignee)

Comment 2

10 years ago
The community ESX host in Amsterdam could probably take a few more VMs but it'll be CPU constrained.  CPU is over 75% busy.

To handle the 6 VMs would very likely require another ESX server (with storage).  The Macs could either be two Mac Minis or a single Xserve running some number of VMs (which depends on what other community needs might come up).

Comment 3

10 years ago
It would be helpful to get some cost quotes on these machines.  Community Giving will need to evaluate the cost to help make the case for leverage.  

MRZ: is it easy to provide a cost for a single Xserve and how many VMs can run here using an "average" amount of the Xserve resources?

Mac Minis, I am guessing, could run about $600-800 apiece, no?
(Assignee)

Comment 4

10 years ago
The ESX hardware is ballpark $8k + software license = $12k.  We typically budget about 20 VMs per ESX host.

Xserve is probably around $6k + Parallels server.  We don't have anything like this in production yet so I haven't any idea how many VMs it could handle.
(Reporter)

Comment 5

10 years ago
If we finally have means to virtualize Mac machines, we could potentially move boxes there that currently run on separate hardware, like the PPC Xserve or mini we're already using right now for SeaMonkey machines.

Comment 6

10 years ago
> Xserve is probably around $6k + Parallels server.  We don't have anything like
> this in production yet so I haven't any idea how many VMs it could handle.

Doesn't MoMo use an Xserve for their Mac buildbots? cc: gozer
Yes, VMWare Fusion on the Apple Xserve supports Mac OS X 10.5
Blocks: 470184

Comment 8

10 years ago
Seth, how is budget situation?

Comment 9

10 years ago
I am going to suggest the following:

Start with 2 VMs (one linux, one windows) and one mac mini.  If MRZ would like to discuss the Xserve with me, that is fine.  One way or another we will get you three machines to start.

After we branch, we will revisit getting the testing machine.

Finally, if SeaMonkey wants to be one of the test pilots for Mac OS virtualization, then I will let MRZ determine the best setup.  Not sure if that is possible right now for IT, but KaiRo has suggested that SeaMonkey would like to do this experiment.
seth, any timeframes on all this from your side?

Comment 11

10 years ago
mrz and i chatted about this.  it seems that we'll go with an xserve so SeaMonkey folks can be one of the groups to test the OS virtualization.

mrz can you confirm this?  is your team ready to work on this?
(Assignee)

Comment 12

10 years ago
Fell off my plate, grabbing.
Assignee: sethb → mrz
(Assignee)

Comment 13

10 years ago
Hardware is on order, will update with ETA when I get it.
Whiteboard: on order
(Assignee)

Updated

10 years ago
Whiteboard: on order → on order, ETA 3/19
(Assignee)

Comment 14

10 years ago
ETA on this XServe is 3/19 via Fed EX # 110911170297076
(Assignee)

Comment 15

9 years ago
Was too aggressive on my ETA.  That ETA reflected hardware delivery, not when they'd be online (we've been blocked on two 10GE cross-connects).  Let's shoot for this Friday.
(Assignee)

Updated

9 years ago
Whiteboard: on order, ETA 3/19 → ETA 04/03
(Assignee)

Updated

9 years ago
Assignee: mrz → phong

Comment 16

9 years ago
The server is installed xserve 10.5.  I will need a copy of Parallel server license to complete the install.

Comment 17

9 years ago
Please proceed and bill the $1248.75 to Community Giving.

Comment 18

9 years ago
I got the license key from Seth.

Comment 19

9 years ago
I'm trying to figure out a way to import the current build ESX templates into Parallels server.  If this doesn't work, then we will have to build these as fresh installs.
(Assignee)

Comment 20

9 years ago
Update - the build scrubbed Windows image converted okay.  The CentOS one failed twice.  Might need to do a fresh image (isn't clear if it's an issue with Parallels supporting CentOS5.0 or something else).

Do you -need- the build reference image?  Could you make a new SeaMonkey ref image?

Can you suggest hostnames for these VMs?
(Assignee)

Comment 21

9 years ago
I can't get the CentOS ref image to import and we don't have an x86_64 ref image that I'm aware of.

Can I install a fresh CentOS5.3 image and let you customize both and make those the ref images?
(Reporter)

Comment 22

9 years ago
(In reply to comment #21)
> I can't get the CentOS ref image to import and we don't have an x86_64 ref
> image that I'm aware of.

I'd really like to be as near as possible to the config also used by the Firefox pools, and esp. to our existing refplatform VMs as all the machines of one platform should go into one generic pool in the future (just like Firefox does) to make maintenance as easy as possible (and all generating completely exchangeable binaries).
(Assignee)

Comment 23

9 years ago
That means a fresh install of CentOS 5.0, both 32 and 64bit and you can install whatever tools are needed before calling them ref images.

My concern is that 5.0 is pretty old, probably filled with bugs.

Did you have hostname suggestions?
(Reporter)

Comment 24

9 years ago
For 64bit, I think we can safely go with 5.3, but for 32bit, my main concern is that binaries produced are exchangeable with binaries produced by the refplatform VMs, so the machines can run in a generic pool.

I don't have hostname suggestions, their target is to serve as slaves for a build pool, can we make us have a similar naming scheme as Firefox slaves have?
(Reporter)

Comment 25

9 years ago
OK, Ben Hearsum tells me their 64bit machine is CentOS 5.0 as well, due to being as close to the refplatform as possible, and given that I want to be as near as possible to Firefox reference stuff, it's probably best to go with 5.0 for all the machines then.
I guess that one 32bit VM set up with that would be best for the moment though if we can easily clone it to the second one when I have gone through all the steps to install and configure everything to make it parallel the refplatform.
I wonder if that strategy would work nicely for the Macs as well. ;-)
With this, the question comes up if we want to keep around such reference images of Linux and Mac on Parallels for possible further cloning (the build team might be interested).
(Assignee)

Comment 26

9 years ago
I grabbed the pre-build VM image from https://wiki.mozilla.org/ReferencePlatforms/Linux-Public and was able to get that imported.  

So I have a win2k3-sp2 ref image and now a CentOS5.0 ref image.  Ben says he has an updated ref image though so I'll import that one as well.  Your current Linux VMs are based off the 01/2008 image.
(In reply to comment #26)
> I grabbed the pre-build VM image from
> https://wiki.mozilla.org/ReferencePlatforms/Linux-Public and was able to get
> that imported.  
> 
> So I have a win2k3-sp2 ref image and now a CentOS5.0 ref image.  Ben says he
> has an updated ref image though so I'll import that one as well.  Your current
> Linux VMs are based off the 01/2008 image.

It's on ftp.m.o now: ftp://ftp.mozilla.org/pub/mozilla/VMs/CentOS5-ReferencePlatform.tar.bz2
(Assignee)

Comment 28

9 years ago
However that new VM differs from the old, it has the same import problems.  For now the only ref Linux VM I've been able to import is the one from 01/2008.
(Assignee)

Comment 29

9 years ago
I now have:

1. Fresh install OSX10.5 Server
2. CentOS 5.0 i386 Ref image (01/2008)
3. Windows 2003 SP2 / VC8 Ref Image

At this point I'll need to clone the OSX10.5 Server image and let you customize it for your build purposes and then declare that as a ref image.

There is no x86_64 CentOS ref image.  Do you want a fresh install or postpone on that one?

The other two platforms can be imaged quickly.
(Assignee)

Comment 30

9 years ago
Login info emailed out of band.  The OSX hosts aren't yet up.

 
> In the ideal case, my vision is to get the following machines:
> - 2 VMs with the Linux centos5 reference image,

cb-seamonkey-linux-01 / 63.245.210.32
cb-seamonkey-linux-02 / 63.245.210.33

> - 2 VMs with the Win2k3 reference image,

cb-seamonkey-win32-01 / 63.245.210.34
cb-seamonkey-win32-02 / 63.245.210.35


> - 1 VMs with the Linux centos5 reference image,

cb-seamonkey-linuxdebug-01 / 63.245.210.36


> - 2 Mac machines (min or xserve, whatever IT is more comfortable with),

cb-seamonkey-osx-01 / 63.245.210.37
cb-seamonkey-osx-02 / 63.245.210.38
(Reporter)

Comment 31

9 years ago
(In reply to comment #29)
> I now have:
> 
> 1. Fresh install OSX10.5 Server
> 2. CentOS 5.0 i386 Ref image (01/2008)
> 3. Windows 2003 SP2 / VC8 Ref Image

Thanks a lot!

> There is no x86_64 CentOS ref image.  Do you want a fresh install or postpone
> on that one?

Yes, a fresh install, please. From what I hear, there's only one other machine with x86_64 as of now.
(Reporter)

Comment 32

9 years ago
Hmm, and I can't reach the machines from cb-jumphost01 right now. :(
(Assignee)

Comment 33

9 years ago
(In reply to comment #32)
> Hmm, and I can't reach the machines from cb-jumphost01 right now. :(

Netmask on that network changed but wasn't changed on cb-jumphost01.  I've changed it now and you should be good to go.
(Assignee)

Updated

9 years ago
Assignee: phong → mrz
(Reporter)

Comment 34

9 years ago
Thanks, I can reach and log into the Linux and Windows VMs now and have brought up the Linux ones to the state of the current refplatform except for scratchbox.

What I didn't realize at first is that we also need extra disks for builds on the machines, from what I know, a 30 GB disk for every machine should do fine, would that be possible?
(Assignee)

Comment 35

9 years ago
Parallels Server doesn't let me hot-add hard drives.  Can you shut those machines down so I can add?
(Assignee)

Comment 36

9 years ago
The two OSX servers are up.  Login info emailed out of band.
(Reporter)

Comment 37

9 years ago
(In reply to comment #35)
> Parallels Server doesn't let me hot-add hard drives.  Can you shut those
> machines down so I can add?

I did halt the Linux machines, I can't find the knob though to shut down the Windows ones.
(Assignee)

Comment 38

9 years ago

> What I didn't realize at first is that we also need extra disks for builds on
> the machines, from what I know, a 30 GB disk for every machine should do fine,
> would that be possible?

Added for each platform.  The OSX images started off with a large disk (64GB) than the others so I didn't add anything there.

At this point all you need is a 64bit CentOS 5.0 install.  And don't update the OSX image - it'll stop booting (still debugging that).

Parallels Server doesn't understand Vlans.  I've ordered a dual-port ethernet card and will need to take a downtime to install it.
(Assignee)

Comment 39

9 years ago
Found the fix, updating the Server software...
Whiteboard: ETA 04/03
(Assignee)

Comment 40

9 years ago
Okay, you can update to 10.5.6.  You should  not update to 10.5.7 until we've confirmed you won't hose your VM.  

When you have the OSX VMs configured shut one down and let me know so I can clone a ref image from it.
(Assignee)

Comment 41

9 years ago
Am building the x86_64 CentOS 5.0 VM off the kickstart @ https://bug419924.bugzilla.mozilla.org/attachment.cgi?id=306233 .
(Reporter)

Comment 42

9 years ago
(In reply to comment #41)
> Am building the x86_64 CentOS 5.0 VM off the kickstart @
> https://bug419924.bugzilla.mozilla.org/attachment.cgi?id=306233 .

That should give me the same base system as in https://wiki.mozilla.org/ReferencePlatforms/Linux-CentOS-5.0 right? If so, probably a good idea to link that kickstart from that doc :)
(Assignee)

Comment 43

9 years ago
cb-seamonkey-linux64-01 / 63.245.209.39

(login is as the other linux hosts)

I attached two additional drives to match the 32bit Linux hosts but left it to you to fdisk & mount.  When you have this VM configured, file a new bug to take clone of it as a 64bit Ref image.

I'm calling this one closed.  If anything breaks, file new bugs please.
Status: NEW → RESOLVED
Last Resolved: 9 years ago
Resolution: --- → FIXED
(Reporter)

Comment 44

9 years ago
(In reply to comment #40)
> When you have the OSX VMs configured shut one down and let me know so I can
> clone a ref image from it.

cb-seamonkey-osx-02 has been shut down for ref imaging, the setup and remaining steps for post-imaging are documented at https://wiki.mozilla.org/User:KaiRo/ParallelsMacOSX (doc can moved to a more "official" place on the wiki as wanted, when doing that, will redirect the current place accordingly).
(Reporter)

Updated

9 years ago
Blocks: 492221
(In reply to comment #30)

> > - 2 VMs with the Linux centos5 reference image,
> 
> cb-seamonkey-linux-01 / 63.245.210.32
> cb-seamonkey-linux-02 / 63.245.210.33
> 
> > - 1 VMs with the Linux centos5 reference image,
> 
> cb-seamonkey-linuxdebug-01 / 63.245.210.36

As well rename this one to 'cb-seamonkey-linux-03' as they're used in a pool of 3 actually.
(Reporter)

Comment 46

9 years ago
(In reply to comment #45)
> > cb-seamonkey-linuxdebug-01 / 63.245.210.36
> 
> As well rename this one to 'cb-seamonkey-linux-03' as they're used in a pool of
> 3 actually.

Nah, doesn't matter, it's already running and set up, and the internal machine names don't matter much.
(Reporter)

Comment 47

9 years ago
(In reply to comment #43)
> cb-seamonkey-linux64-01 / 63.245.209.39
> 
> (login is as the other linux hosts)
> 
> I attached two additional drives to match the 32bit Linux hosts but left it to
> you to fdisk & mount.  When you have this VM configured, file a new bug to take
> clone of it as a 64bit Ref image.

Hmm, I somehow overlooked that we want to do a ref image of it when setting it up, so it has the seabld user and SeaMonkey builder passwords now. Should I neutralize this somehow before we take the ref image?
(Assignee)

Comment 48

9 years ago
Up to you - you're likely the only user of this ref image in the near term.
(Reporter)

Updated

9 years ago
Blocks: 492527
(Reporter)

Comment 49

9 years ago
OK, I changed to root password to what the Mac ref image has and stripped the ssh keys from the seabld user, I guess that's generic enough that anyone can consume the image. Filed bug 492527 for imaging.
(Reporter)

Comment 50

9 years ago
verifying, we have all those shiny new VMs, thanks a lot to everyone involved!
Status: RESOLVED → VERIFIED
(Reporter)

Updated

9 years ago
Blocks: 485820
(Reporter)

Updated

9 years ago
Blocks: 493454
You need to log in before you can comment on or make changes to this bug.