The default bug view has changed. See this FAQ.

Promote SUMO to new DB cluster

RESOLVED FIXED

Status

mozilla.org Graveyard
Server Operations
--
minor
RESOLVED FIXED
8 years ago
2 years ago

People

(Reporter: mrz, Assigned: justdave)

Tracking

Bug Flags:
needs-downtime +

Details

(Reporter)

Description

8 years ago
Tracking bug to promote SUMO to a non-C level DB cluster.  

Dave, is this A or B worthy?  Can we shoot for Thursday?
(Reporter)

Updated

8 years ago
Assignee: server-ops → justdave
CCing morgamic since he was asking about tracking bugs earlier.

My personal opinion is that this is worthy of A-level SLA now, due to the fact that it's direct-linked from the Help menu in a shipping product, and it's enduser-facing.  It's not the end of the world if it's down, but it's a really bad user experience for someone frustrated and trying to get help.

IIRC I questioned it being put on the C cluster when it was first set up, but was told "it's not user-facing yet, we can move it later".  Guess later never came (until now).

Our B-level SLA for databases indicates that we can tolerate small amounts of downtime, so maybe it'll fit there.  The existing B cluster is mostly bored these days.  If we put SUMO there, it would likely generate most of the traffic for it.  We wouldn't need new hardware to do that.  If we go with A-level, we'll need two more hardware boxes to put this on.  Blades with 8GB RAM would be fine.
(Reporter)

Comment 2

8 years ago
I need CPU count and memory count.  Your wiki page has three listed.

Comment 3

8 years ago
I guess this didn't happen last night - do we have a revised ETA?  We are pushing 1.0 with requisite PR on Tuesday, so it would be nice to have the extra grunt.
(Reporter)

Comment 4

8 years ago
dmoore, can you update with an ETA from CRG on when they'll have the 10GE links up?  

We're gated on that right now.
(Reporter)

Comment 5

8 years ago
Hardware allocated.  All 4GB, single quad-core Xeon.  Is it reasonable to schedule this for a Tuesday night cut-over?

https://inventory.mozilla.org/systems/show/961
https://inventory.mozilla.org/systems/show/962
https://inventory.mozilla.org/systems/show/963
(In reply to comment #5)
> Hardware allocated.  All 4GB, single quad-core Xeon.

As per comment #1, need 8GB RAM, no?
I suspect 4GB will be fine, actually.  It's all MyISAM stuff, they aren't using InnoDB yet.  The huge amounts of RAM are typically used for InnoDB caching.

The servers that it's running on now are all 4GB for that matter, and there's a lot of other stuff on that cluster with it right now.
(Reporter)

Comment 8

8 years ago
We're going to tentatively shoot for Tuesday night for a cut over.  Can this go with your 1.0 release or should we push to Thursday?
Flags: needs-downtime+

Comment 9

8 years ago
As long as 1.0 can still be released in case there are problems with the DB stuff, that sounds good to me. (1.0 is a Q1 goal.) Thanks!
I'm ok with it except for one thing...we *should* have some InnoDB tables, I
ran a script to convert the DB and Jeremy ran it on prod prior to the Fx 3
release.  If they're not InnoDB now then a bad thing has occurred.

Anyway, we should get the 8GB regardless, because if we're not on InnoDB now
then we need to convert the tables that we can.  (We can't bulk convert as we
have some reliance on MyISAM full text indexing which will however go away
soon.)
(Reporter)

Comment 11

8 years ago
dave - does 8GB block this or can we roll with 4GB and take a window later to add memory?
4GB will be a perf improvement over what it's got now, because it's sharing 4GB with 67 other databases right now.
Hey, 4GB sounds okay for now but when it comes to RAM I'd prefer to overcompensate so please try 8GB or 16GB sooner than later.
(Reporter)

Comment 14

8 years ago
So for clarification we'll start with 4GB and do a memory upgrade shortly.
Okay -- also keep in mind that there are InnoDB tables in that database.  Laura may have more info, but not all the traffic is hitting MyISAM.
Looks like the sessions table is InnoDB, that's it.  Also, I notice every one of the tables has latin1 as the default charset.  Thought this thing was new enough to be using utf8, but who knows... :)
That's the mysql default so means two things:
1) it probably wasn't explicitly defined as utf8 in the SUMO sql
2) the defaults for the server are not utf8
(Reporter)

Comment 18

8 years ago
Done, lastnight.
Status: NEW → RESOLVED
Last Resolved: 8 years ago
Resolution: --- → FIXED
Product: mozilla.org → mozilla.org Graveyard
You need to log in before you can comment on or make changes to this bug.