560619 - Need a new home for brasstacks results Db and webserver

Status: RESOLVED WORKSFORME

(mozilla.org Graveyard :: Server Operations, task)

Description

[cmtalbert]

Our current sm-brasstacks VM is running beyond capacity. It doesn't look like the ESXi machine is overloaded per se, but the experience of the users of the web tools and databases housed on that machine are reporting serious problems getting to the data.

I think it may be time to split this guy off and make him a real machine.  This machine (and the tools on it) are becoming our results reporting server for many of the tools that we are developing and completing.  Currently it houses:
* Grafx Bot test results
* Topfails web tool
* Automated Mozmill QA Results

We are expanding each of these tools as well as creating new ones which will be talking to the systems on this machine.  We need this machine to be able to scale better to meet the needs of a web/database server.

It should be:
* 64 bit Linux 
* Quad core
* 4Gb RAM
* 1T disk

It should still be available publicly as brasstacks.mozilla.com.

We're looking for IT help here on these points:
* Should it be a VM or a stand-alone box?  It doesn't seem like it'd have to be an amazing box, but it definitely needs much more resources than it currently has.
* What OS should it use?  I'm doing some experiments with my own Vms trying to answer that question myself: I'm currently looking at Ubunutu Server 64 bit 9.10 and RHEL 64 bit, 5.3.  But, I'd cede to your expertise on this point.

Comment 1

[Henrik Skupin [:whimboo][⌚️UTC+1]]

(In reply to comment #0)
> machine (and the tools on it) are becoming our results reporting server for
> many of the tools that we are developing and completing.  Currently it houses:
> * Automated Mozmill QA Results

Regarding the Mozmill tests I have to add that we are creating a cloud testing extension right now, which will allow any user to run Mozmill tests by a single click and it will send the report to brasstacks too. That means we will have to cover a lot more results per day as we currently do. Further we also plan to extend our in-house Mozmill tests by running tests against add-ons.

Comment 2

[Shyam Mani [:fox2mike]]

(In reply to comment #0)
> We're looking for IT help here on these points:
> * Should it be a VM or a stand-alone box?  It doesn't seem like it'd have to be
> an amazing box, but it definitely needs much more resources than it currently
> has.

We could try bumping up the resources on the current VM and see if that helps? If it doesn't, I guess we can be sure we'd have to go to hardware. 

> * What OS should it use?  I'm doing some experiments with my own Vms trying to
> answer that question myself: I'm currently looking at Ubunutu Server 64 bit
> 9.10 and RHEL 64 bit, 5.3.  But, I'd cede to your expertise on this point.

IT usually goes with RHEL 5.x but we just got our hands on 6 beta ;) That'll be interesting too.

Comment 3

[Henrik Skupin [:whimboo][⌚️UTC+1]]

(In reply to comment #2)
> We could try bumping up the resources on the current VM and see if that helps?
> If it doesn't, I guess we can be sure we'd have to go to hardware. 

For Mozmill we will have a lot more traffic soon. I would feel better when we could get our own dedicated machine and can also get rid of CentOS in the same step.

Comment 4

[Shyam Mani [:fox2mike]]

mrz,

Can I bring this up in PHX? I see a bunch of spare machines on thermo-b10 with 6 and 12 GB of RAM, could put one of the sixers on this?

Comment 5

[matthew zeier [:mrz]]

Depends what sort of access this needs.  There isn't any user VPN in Phoenix and no QA environment there.  I'd lean towards SJC.

Comment 6

[cmtalbert]

(In reply to comment #5)
> Depends what sort of access this needs.  There isn't any user VPN in Phoenix
> and no QA environment there.  I'd lean towards SJC.
As we're still developing many of the applications on this box, I'd prefer to keep it in SJC in order to maintain our ability to VPN/ssh into it.  I used VIC to bump the ram up to 2Gb (was 768Mb), so that does seem to helping a bit.  I didn't go too much higher because I'm not sure how much I can safely overload qm-vmware02's memory before it becomes a problem with the other machines.

Comment 7

[matthew zeier [:mrz]]

> I
> didn't go too much higher because I'm not sure how much I can safely overload
> qm-vmware02's memory before it becomes a problem with the other machines.

IIRC, you can over subscribe memory.  If it becomes an issue, we'll add memory to that  ESX host.

Comment 8

[Shyam Mani [:fox2mike]]

Clint, have you tried boosted RAM further? Still need physical hardware for this?

Comment 9

[cmtalbert]

(In reply to comment #8)
> Clint, have you tried boosted RAM further? Still need physical hardware for
> this?
Sorry, forgot to reply to mrz.  I boosted the memory and it is running ok.  Diskspace use will be an issue eventually, but I think with us moving the sm-purify box to physical hardware, we can increase the diskspace here as needed so we should be ok.

I'll close this as wfm now.  Thanks for following up w/me.