343292 - Tracking for new machine for tinderbox/bonsai/[possibly] lxr

Status: RESOLVED FIXED

(mozilla.org Graveyard :: Server Operations, task)

Description

[J. Paul Reed [:preed]]

This is a tracking bug for getting a new home for bonsai, tinderbox, and possibly lxr. This is timely, since I guess mecha is going away in a few weeks?

Anyway, bonsai and tinderbox need to be on the same machine.

lxr doesn't, but it might be nice for ease-of-administration if it is.

Due to the performance requirements of tinderbox and bonsai (quite high, due to people pounding on tinderbox, showbuilds.cgi not being particularly efficient, lots of network traffic for incoming logs from tbox machines and outgoing logs from people requesting... well... logs), it's unclear whether or not a VM will suffice. It might.

Comment 1

[Mike Schroepfer]

Note we should put the storage on the Netapp since we seem to keep running out of space for Tinderbox logs and those logs are in general very useful

Comment 2

[matthew zeier [:mrz]]

(In reply to comment #1)
> Note we should put the storage on the Netapp since we seem to keep running out
> of space for Tinderbox logs and those logs are in general very useful

Do you have an idea of how much initial space you'll need?  I assume NFS is sufficient (vs. iscsi)?

Comment 3

[J. Paul Reed [:preed]]

(In reply to comment #2)

> Do you have an idea of how much initial space you'll need?

Current Tinderbox installation is using ~48 gigs, according to du.

We'd likely want at least double that.

> I assume NFS is sufficient (vs. iscsi)?

Given tinderbox's loads, NFS's... "issues," and the fact that I'm willing to bet that tinderbox assumes it's *not* running on NFS, we probably don't want NFS.

You might want to coordinate with justdave on this; we was working on a new tinderbox installation on mecha last I saw.

Were bonsai/tinderbox going to be put on a VM?

Comment 4

[Dave Miller [:justdave]]

Yeah, 100 GB of disk space at a minimum, 150GB might be nice if it's available.

I was personally kinda leaning towards NFS, but only because of being able to share the data between the old and new installs during the transition.  And I suppose someday we might want to load-balance the app, but probably not for a few years, so iSCSI would probably be fine.

Comment 5

[matthew zeier [:mrz]]

Box built, over to Dave.

Comment 6

[Dave Miller [:justdave]]

We've got dm-webtools01, working on setting it up now, hope to have this up as a staging environment for testing later tonight.

Comment 7

[Dave Miller [:justdave]]

we had load issues with dm-webtools01 (which is a vm) handling the volume of mail coming in from the tinderboxes, going to try again on dm-webtools02 (a physical DL360) in the next day or so.

The minor webtools and stuff that doesn't need DB or direct bonsai access will probably still go on dm-webtools01.

Comment 8

[Dave Miller [:justdave]]

OK, the box is there, and everything seems to work.  The apps aren't live yet, but those appear to have their own bugs (dependencies of this one, go figure ;)