484891 - Promote SUMO to new DB cluster

Status: RESOLVED FIXED

(mozilla.org Graveyard :: Server Operations, task)

Description

[matthew zeier [:mrz]]

Tracking bug to promote SUMO to a non-C level DB cluster.  

Dave, is this A or B worthy?  Can we shoot for Thursday?

Comment 1

[Dave Miller [:justdave]]

CCing morgamic since he was asking about tracking bugs earlier.

My personal opinion is that this is worthy of A-level SLA now, due to the fact that it's direct-linked from the Help menu in a shipping product, and it's enduser-facing.  It's not the end of the world if it's down, but it's a really bad user experience for someone frustrated and trying to get help.

IIRC I questioned it being put on the C cluster when it was first set up, but was told "it's not user-facing yet, we can move it later".  Guess later never came (until now).

Our B-level SLA for databases indicates that we can tolerate small amounts of downtime, so maybe it'll fit there.  The existing B cluster is mostly bored these days.  If we put SUMO there, it would likely generate most of the traffic for it.  We wouldn't need new hardware to do that.  If we go with A-level, we'll need two more hardware boxes to put this on.  Blades with 8GB RAM would be fine.

Comment 2

[matthew zeier [:mrz]]

I need CPU count and memory count.  Your wiki page has three listed.

Comment 3

[Laura Thomson :laura]

I guess this didn't happen last night - do we have a revised ETA?  We are pushing 1.0 with requisite PR on Tuesday, so it would be nice to have the extra grunt.

Comment 4

[matthew zeier [:mrz]]

dmoore, can you update with an ETA from CRG on when they'll have the 10GE links up?  

We're gated on that right now.

Comment 5

[matthew zeier [:mrz]]

Hardware allocated.  All 4GB, single quad-core Xeon.  Is it reasonable to schedule this for a Tuesday night cut-over?

https://inventory.mozilla.org/systems/show/961
https://inventory.mozilla.org/systems/show/962
https://inventory.mozilla.org/systems/show/963

Comment 6

[Reed Loden [:reed]]

(In reply to comment #5)
> Hardware allocated.  All 4GB, single quad-core Xeon.

As per comment #1, need 8GB RAM, no?

Comment 7

[Dave Miller [:justdave]]

I suspect 4GB will be fine, actually.  It's all MyISAM stuff, they aren't using InnoDB yet.  The huge amounts of RAM are typically used for InnoDB caching.

The servers that it's running on now are all 4GB for that matter, and there's a lot of other stuff on that cluster with it right now.

Comment 8

[matthew zeier [:mrz]]

We're going to tentatively shoot for Tuesday night for a cut over.  Can this go with your 1.0 release or should we push to Thursday?

Comment 9

[David Tenser [:djst]]

As long as 1.0 can still be released in case there are problems with the DB stuff, that sounds good to me. (1.0 is a Q1 goal.) Thanks!

Comment 10

[Laura Thomson :laura]

I'm ok with it except for one thing...we *should* have some InnoDB tables, I
ran a script to convert the DB and Jeremy ran it on prod prior to the Fx 3
release.  If they're not InnoDB now then a bad thing has occurred.

Anyway, we should get the 8GB regardless, because if we're not on InnoDB now
then we need to convert the tables that we can.  (We can't bulk convert as we
have some reliance on MyISAM full text indexing which will however go away
soon.)

Comment 11

[matthew zeier [:mrz]]

dave - does 8GB block this or can we roll with 4GB and take a window later to add memory?

Comment 12

[Dave Miller [:justdave]]

4GB will be a perf improvement over what it's got now, because it's sharing 4GB with 67 other databases right now.

Comment 13

[Michael Morgan [:morgamic]]

Hey, 4GB sounds okay for now but when it comes to RAM I'd prefer to overcompensate so please try 8GB or 16GB sooner than later.

Comment 14

[matthew zeier [:mrz]]

So for clarification we'll start with 4GB and do a memory upgrade shortly.

Comment 15

[Michael Morgan [:morgamic]]

Okay -- also keep in mind that there are InnoDB tables in that database.  Laura may have more info, but not all the traffic is hitting MyISAM.

Comment 16

[Dave Miller [:justdave]]

Looks like the sessions table is InnoDB, that's it.  Also, I notice every one of the tables has latin1 as the default charset.  Thought this thing was new enough to be using utf8, but who knows... :)

Comment 17

[Michael Morgan [:morgamic]]

That's the mysql default so means two things:
1) it probably wasn't explicitly defined as utf8 in the SUMO sql
2) the defaults for the server are not utf8

Comment 18

[matthew zeier [:mrz]]

Done, lastnight.