477183 - Add additional database slaves for AMO, Bouncer, and Bugzilla

Status: RESOLVED FIXED

(mozilla.org Graveyard :: Server Operations, task)

Description

[matthew zeier [:mrz]]

During release periods, bouncer needs at least two databases to handle load.  Need to add a third to be able to withstand a failure of any one during release.

Comment 1

[Dave Miller [:justdave]]

got hardware for me?

Comment 2

[Dave Miller [:justdave]]

We need additional slaves for AMO and Bugzilla, as well.  I have the designated hot spares assigned to those two right now, and I'd like to have spares available for emergencies again.

Comment 3

[Dave Miller [:justdave]]

I've snagged the former mrdb05 box for the new AMO slave, since it's the same type of hardware the rest of the AMO slaves are using.  None of the other free hardware we have at the moment is appropriate for the other two though.  Supposedly we have some new blades getting racked any day now, I'll wait for those.

Comment 4

[Dave Miller [:justdave]]

FYI, just discovered the existing Bugzilla slave has 16 GB of RAM (and is actually using it, 14GB physical in use at the moment - the temporary box I have helping out in that pool right now only has 8 and it's swapping).  So we'll need to make sure we have that much available in one of the boxes for that.

Comment 5

[Dave Miller [:justdave]]

This is now blocker status until the new bouncer slave is up.  This one in particular must be done before the FF 3.0.7 release.  Getting the additional AMO slave up before the FF 3.0.7 release will be a bonus.

Comment 6

[matthew zeier [:mrz]]

Can you take one of these machines temporarily?  There are a couple there you could spin up in short order.

Comment 7

[Dave Miller [:justdave]]

I have one allocated for bouncer already.  Was hoping for a couple of the new blades for AMO and Bugzilla.

Comment 8

[Dave Miller [:justdave]]

Hmm, I take that back, AMO is the one I already had hardware for.  I need one for bouncer still.  I'll snag one of those blades for Bouncer.

Comment 9

[Dave Miller [:justdave]]

tm-amo01-slave01 and tm-bouncer01-slave03 are now online.  The bouncer slave is on crummy hardware, but it'll survive for now. :)

Comment 10

[Dave Miller [:justdave]]

Just to clarify (since someone asked), by "crummy" I meant non-redundant.  It's using one of the blades from the test lab chassis that doesn't have redundant power supplies/NICs/etc.  It's otherwise got the usual slate of processor and RAM so it should keep up just fine.  Considering that it's in a pool with other mostly-identical slaves anyway, that probably doesn't really matter.

Comment 11

[Dave Miller [:justdave]]

oh, we're better off than I thought it was...

20:27:48 < mrz> hey dave, the BL260c vs. the BL460c - the 260 is sata
20:27:50 < mrz> only difference
20:28:12 < mrz> the chassis it's in, however, differs from the c7000 chassis in
                that it has only one pass-through module and is an 8 bay chassis
                vs. 16
20:28:19 < mrz> but fully redundant power

Comment 12

[matthew zeier [:mrz]]

Phong is going to check inventory to see if we have memory in stock.

Comment 13

[Phong Tran [:phong]]

We have 1 pack of 2x2GB.  Go ahead and order 12 GB more.

Comment 14

[matthew zeier [:mrz]]

Bugzilla: https://inventory.mozilla.org/systems/show/957

Comment 15

[matthew zeier [:mrz]]

What do you need for AMO & bouncer?

Comment 16

[Dave Miller [:justdave]]

(In reply to comment #15)
> What do you need for AMO & bouncer?

Nothing, unless you want to get the tm-bouncer01-slave03 off the test lab chassis.

Comment 17

[matthew zeier [:mrz]]

I do.  That's lab hardware, not meant to be long term production.

Take https://inventory.mozilla.org/systems/show/965 (and rename if needed).

Comment 18

[Dave Miller [:justdave]]

I forgot to mention the second Bugzilla slave is up and in the slave pool as of Thursday or Friday last week.  Haven't moved bouncer01-slave03 to the new hardware yet.

Comment 19

[Dave Miller [:justdave]]

ok, bouncer01-slave03 has been moved.  This is done. :)