Closed Bug 484891 Opened 12 years ago Closed 11 years ago

Promote SUMO to new DB cluster

Categories

(mozilla.org Graveyard :: Server Operations, task)

All
Other
task
Not set
minor

Tracking

(Not tracked)

RESOLVED FIXED

People

(Reporter: mrz, Assigned: justdave)

Details

Tracking bug to promote SUMO to a non-C level DB cluster.  

Dave, is this A or B worthy?  Can we shoot for Thursday?
Assignee: server-ops → justdave
CCing morgamic since he was asking about tracking bugs earlier.

My personal opinion is that this is worthy of A-level SLA now, due to the fact that it's direct-linked from the Help menu in a shipping product, and it's enduser-facing.  It's not the end of the world if it's down, but it's a really bad user experience for someone frustrated and trying to get help.

IIRC I questioned it being put on the C cluster when it was first set up, but was told "it's not user-facing yet, we can move it later".  Guess later never came (until now).

Our B-level SLA for databases indicates that we can tolerate small amounts of downtime, so maybe it'll fit there.  The existing B cluster is mostly bored these days.  If we put SUMO there, it would likely generate most of the traffic for it.  We wouldn't need new hardware to do that.  If we go with A-level, we'll need two more hardware boxes to put this on.  Blades with 8GB RAM would be fine.
I need CPU count and memory count.  Your wiki page has three listed.
I guess this didn't happen last night - do we have a revised ETA?  We are pushing 1.0 with requisite PR on Tuesday, so it would be nice to have the extra grunt.
dmoore, can you update with an ETA from CRG on when they'll have the 10GE links up?  

We're gated on that right now.
Hardware allocated.  All 4GB, single quad-core Xeon.  Is it reasonable to schedule this for a Tuesday night cut-over?

https://inventory.mozilla.org/systems/show/961
https://inventory.mozilla.org/systems/show/962
https://inventory.mozilla.org/systems/show/963
(In reply to comment #5)
> Hardware allocated.  All 4GB, single quad-core Xeon.

As per comment #1, need 8GB RAM, no?
I suspect 4GB will be fine, actually.  It's all MyISAM stuff, they aren't using InnoDB yet.  The huge amounts of RAM are typically used for InnoDB caching.

The servers that it's running on now are all 4GB for that matter, and there's a lot of other stuff on that cluster with it right now.
We're going to tentatively shoot for Tuesday night for a cut over.  Can this go with your 1.0 release or should we push to Thursday?
Flags: needs-downtime+
As long as 1.0 can still be released in case there are problems with the DB stuff, that sounds good to me. (1.0 is a Q1 goal.) Thanks!
I'm ok with it except for one thing...we *should* have some InnoDB tables, I
ran a script to convert the DB and Jeremy ran it on prod prior to the Fx 3
release.  If they're not InnoDB now then a bad thing has occurred.

Anyway, we should get the 8GB regardless, because if we're not on InnoDB now
then we need to convert the tables that we can.  (We can't bulk convert as we
have some reliance on MyISAM full text indexing which will however go away
soon.)
dave - does 8GB block this or can we roll with 4GB and take a window later to add memory?
4GB will be a perf improvement over what it's got now, because it's sharing 4GB with 67 other databases right now.
Hey, 4GB sounds okay for now but when it comes to RAM I'd prefer to overcompensate so please try 8GB or 16GB sooner than later.
So for clarification we'll start with 4GB and do a memory upgrade shortly.
Okay -- also keep in mind that there are InnoDB tables in that database.  Laura may have more info, but not all the traffic is hitting MyISAM.
Looks like the sessions table is InnoDB, that's it.  Also, I notice every one of the tables has latin1 as the default charset.  Thought this thing was new enough to be using utf8, but who knows... :)
That's the mysql default so means two things:
1) it probably wasn't explicitly defined as utf8 in the SUMO sql
2) the defaults for the server are not utf8
Done, lastnight.
Status: NEW → RESOLVED
Closed: 11 years ago
Resolution: --- → FIXED
Product: mozilla.org → mozilla.org Graveyard
You need to log in before you can comment on or make changes to this bug.