522112 - RAM upgrades for AMO database servers

Status: RESOLVED FIXED

(mozilla.org Graveyard :: Server Operations, task)

Description

[Dave Miller [:justdave]]

Per recommendation from Percona, our addons DB would perform best with at least 10 GB of innodb_buffer_pool and at least a few gigs of index cache.  To achieve this, we need to get all of the servers up to 16 GB of RAM.  Currently tm-amo01-slave01 has 8 GB of RAM, and the rest of the cluster (including the master) only has 4 GB.
                                                             -------RAM--------
database           model    CPU                              max  slots current
tm-amo01-master01  DL360G5  2 x 2.66GHz Xeon 5150 dual core  32GB   8   4 x 1GB
tm-amo01-slave01   DL360G5  2 x 2.66GHz Xeon 5150 dual core  32GB   8   8 x 1GB
tm-amo01-slave02   BL460cG1 2 x 2.5GHz Xeon L5420 dual core  32GB   8   4 x 1GB
tm-amo01-slave03   BL460cG1 2 x 2.5GHz Xeon L5420 dual core  32GB   8   4 x 1GB

Comment 1

[matthew zeier [:mrz]]

phong, you said we had a lot of spare memory - can you use as much of that as possible to get here?

Comment 3

[Phong Tran [:phong]]

I'll get the 2 BL460cG1 up to 16GB.  I'll see if we have spare RAM for the DL360G5.

Can we take these down during the day to add RAM?

Comment 4

[Phong Tran [:phong]]

tm-amo01-slave02   BL460cG1 2 x 2.5GHz Xeon L5420 dual core  32GB   8   4 x 2GB
tm-amo01-slave03   BL460cG1 2 x 2.5GHz Xeon L5420 dual core  32GB   8   4 x 2GB

Comment 5

[Phong Tran [:phong]]

tm-amo01-slave01   DL360G5  2 x 2.66GHz Xeon 5150 dual core  32GB   8   8 x 2GB

Comment 6

[Phong Tran [:phong]]

tm-amo01-master01 needs downtime.

Comment 7

[matthew zeier [:mrz]]

Any desire to do this tonight along with the AMO update?

Comment 8

[Wil Clouser [:clouserw]]

There is a desire from my side. :)  What kind of downtime are we talking about?

Comment 9

[matthew zeier [:mrz]]

this may be blocked on IT availability to be there this evening... phong?  aravind?  zandr?

Comment 10

[Dave Dash [:davedash, :dd] (assign all bugs to mbrandt)]

Can we promote one of the upgraded slaves to master, and upgrade the old master at a later time?

Comment 11

[Phong Tran [:phong]]

I have a 7 PM soccer game tonight so it might be bad for me.

Comment 12

[Wil Clouser [:clouserw]]

Is this happening tonight?

Comment 13

[Phong Tran [:phong]]

tm-amo01-master01 done.

Comment 14

[Jeff Balogh (:jbalogh)]

(In reply to comment #0)
> Per recommendation from Percona, our addons DB would perform best with at least
> 10 GB of innodb_buffer_pool and at least a few gigs of index cache.  To achieve
> this, we need to get all of the servers up to 16 GB of RAM.

Do we need another bug to adjust the buffer pool?