416469 - setup unit/mochi/crash/reftest buildbots for SeaMonkey

Status: RESOLVED FIXED

(SeaMonkey :: Build Config, defect, P3)

Description

[Robert Kaiser]

Currently, SeaMonkey doesn't have many unit/mochi/crash/reftests specific to suite code, but it's not very compelling to write such tests if they are not run automatically.
The Linux tinderbox on the SeaMonkey waterfall is running TUint now, but that makes build cycles longer, and tinderbox' reporting of those tests to the waterfall is not ideal.

It would be cool to have buildbot slaves for SeaMonkey which are running those tests and can monitor quality of suite code (given that we also create more tests following that exposure).

Comment 1

[Justin Wood (:Callek)]

for the record there was also a Tunit mailnews, related crash... that was platform specific (in this case), that would have been caught much sooner had there been a machine running win Tunit at the least.

Comment 2

[Robert Kaiser]

dmose from the mail team told me that joduinn could possibly help with getting such testing boxes for SeaMonkey - is that correct?

John, any chance we can get something going there?

Comment 3

[John O'Duinn [:joduinn] (please use "needinfo?" flag)]

(Thanks kairo for offline ping on this)

Reassigning so this will show up on triage. From discussions with Kairo on irc just now:

1) They'd like 2 VMs (1 linux, 1 win32). Is there any room left on the community ESX machine to create new VMs?
2) They'd like 1 mac. Because its the trunk/1.9 branch, ideally it should be an intel mac xserve, the same as equivalent firefox branch.

Comment 4

[Robert Kaiser]

If money for machines may be an issue here, we possibly could take SeaMonkey-directed donation money from MoFo for this, CCing David Boswell for this.

Comment 5

[David Boswell]

Re comment #4, if this is an issue of money then I think you certainly could use the SeaMonkey directed donations for this.  Kairo, feel free to submit a proposal to me and/or Frank if you want to move forward with this option.

Comment 6

[John O'Duinn [:joduinn] (please use "needinfo?" flag)]

cc-ing Seth, as he had some suggestions after meeting with Justin and myself.

Comment 7

[Mike Shaver (:shaver emeritus)]

Does the Mac need to be an xserve?  Test latency bounds are typically proportionate to commit-frequency: the more SeaMonkey-specific checkins there are, the higher frequency the sampling needs to be to aid in regression hunting.

An xserve is a big bite, and can't be as easily partitioned later for split use due to lack of OS X virtualization.  I think we can do a mini and the VMs without too much trouble; if that runs out of capacity, we can look at options for scaling and figure out how the cost might be apportioned.

Comment 8

[Seth Bindernagel]

We're also investigating partitioning the l10n box in our Amsterdam colo to place VMs there for the SeaMonkey project.  That box is an underused resource that we might could make available for something like this.  I will report back soon.

Comment 9

[Robert Kaiser]

The SeaMonkey project is happy with any solution that gives us machines on all three platforms that perform the unit/mochi/crash/retest suites. So from our POV, we can go with either solution on the Mac side, whatever seems better.

Comment 10

[Robert Kaiser]

Seth, I read in your blog the VMs were already created? Or is this effort still underway?

Comment 11

[Seth Bindernagel]

Shaver and I have approved it, that's why I blogged it.  The creation of the VMs is still underway.  MRZ will help with that.  I'll check with him to see what he needs from me.

Comment 12

[matthew zeier [:mrz]]

I asked for VM specs in bug #423868 - still waiting on that.  

And by Linux, are you talking about CentOS or Ubuntu?

Comment 13

[Seth Bindernagel]

Kairo, can you respond to MRZ?

Comment 14

[Robert Kaiser]

From what I see, FF is running the unit test machine on CentOS, so I it'd probably best to match that.

Comment 15

[matthew zeier [:mrz]]

10GB disk good enough?  512MB RAM? Just a fresh install of CentOS? 

Comment 16

[Robert Kaiser]

Our best reference are probably the FF unit test boxes, but this sounds basically fine for me.

Comment 17

[matthew zeier [:mrz]]

Build folk - is there a CentOS reference image I should just clone?

Comment 18

[John O'Duinn [:joduinn] (please use "needinfo?" flag)]

For FF unittests, we use the following VMs:

win2k3sp2-vc8tools-ref-vm
CentOS-5.0-ref-tools-vm

Comment 19

[Robert Kaiser]

Those are the most current refplatforms, right?
I just realized that IIRC unit test boxen compile builds theirselves, so they probably should just have the same ref platforms as the tinderboxen.

Comment 20

[matthew zeier [:mrz]]

Going to take a while (long while) to copy these ref images out to Amsterdam.  Work in progress...

Comment 21

[matthew zeier [:mrz]]

Robert, can you give me hostnames?  

Other VMs are something like sea-mozbuild-win32-tbox and sea-newref-linux-tbox.

Comment 22

[Robert Kaiser]

I think it's best to adhere to the naming that FF unit test boxes use, so something like sea-qm-centos5-01 and sea-qm-win2k3-01 makes most sense to me.

Comment 23

[matthew zeier [:mrz]]

VMs are cloning.  IP info below:

cn-sea-qm-centos5-01.nl.mozilla.org / 63.245.212.102
cn-sea-qm-win2k3-01.nl.mozilla.org / 63.245.212.103

Netmask: 255.255.255.128
Gateway: 63.245.212.1

root@cn-rhino01:~# cat /etc/resolv.conf 
search mozilla.org
nameserver 63.245.212.5
nameserver 208.67.222.222
nameserver 208.67.220.220

Comment 24

[matthew zeier [:mrz]]

Both boxes are up.

Comment 25

[Nick Thomas [:nthomas] (UTC+12)]

If the machines aren't QA managed or in MPT, should qm really be in hostname ?

Comment 26

[Robert Kaiser]

If we shouldn't have "qm" there, then we should replace it with "test" or such.

Additionally, I guess the next step is getting buildbot running tests on the VMs and reporting to tinderbox, who can help me get that up and running?

Comment 27

[Robert Kaiser]

Following advice of coop, over to server ops for the next step:

Could you please set up some way so I can access those VMs? Our tinderboxen run on the "seabld" user, maybe that would also be a good choice here, but I'm not sure how such test boxes are usually set up.

Comment 28

[matthew zeier [:mrz]]

cn-sea-qm-centos5-01.nl.mozilla.org was missing a default route, it's reachable now.

Comment 29

[matthew zeier [:mrz]]

Per IRC convo, I think this is fixed. 

Comment 30

[Robert Kaiser]

It's not fixed, as the buildbots are not yet reporting anywhere ;-)

I can access the machines and I have some basic info for setting them up, I may still need some handholding or such for getting everything running well, but for now I have enough work with installing buildbot and configuring master and slaves.

Over to SeaMonkey/Buildconfig and me for getting everything up and running - thanks for creating the machines and setting them up far enough that I can take over!

We still have no mac, but I think we should probably do a followup for that once Linux and Windows work.

Comment 31

[Robert Kaiser]

Attachment: patch for adding buildbot configs to tree

I'd like to have our configs added to the tree just like the files for FF are, so here's the current set I have on the master (the Linux slave runs and gets reported correctly on tinderbox).
Rob, could you please review this?

Comment 32

[Rob Campbell [:rc] (:robcee)]

Comment on attachment 318365 [details] [diff] [review]
patch for adding buildbot configs to tree

other than some missing newlines in mozbuild.py, and waterfall.css, this looks good to me.

Comment 33

[Robert Kaiser]

OK, the master and both Linux and Windows slaves are running and mostly working (open issues are a mochitest focus problem ajschult has a patch for and some know issues on browser tests).

Comment 34

[Robert Kaiser]

Attachment: final patch for adding buildbot configs to tree

This is the final config with which I got everything working in a mode normal enough that we can move off MozillaTest on to SeaMonkey-Ports (will only move to main SeaMonkey page when everything's green, see dependencies), it's mainly Windows path fixes compared to the first patch, carrying over r=rcampbell.

Comment 35

[Robert Kaiser]

I consider this fixed for now, even though not all tests pass yet, but the Linux/Windows boxen are up and running, everything else is filed in different bugs.