Closed Bug 713092 Opened 14 years ago Closed 14 years ago

tm-b01-master01 needs more than 8G RAM

Categories

(Data & BI Services Team :: DB: MySQL, task)

Product:
Data & BI Services Team
Component:
DB: MySQL
Platform:
x86
macOS
Type:
task
Priority:
Not set
Severity:
major

Tracking

(Not tracked)

Status:
RESOLVED FIXED

People

(Reporter: scabral, Assigned: phong)

References

(
URL
)

Details

(Whiteboard: [buildduty][outage])

There are 20 dbs, over 60G of data, and only 8G on the machine itself. Can we get more RAM installed?
raising to major as the mysql issues have been a cause of outages and tree closures for almost two weeks running
Severity: normal → major
Whiteboard: [buildduty][outage]
Blocks: 712830
Blocks: 712988
Blocks: 712986
Assignee: server-ops-database → dgherman
zandr will go to MPT to attempt some RAM upgrades.
No longer blocks: 712988
12:29:58 zandr: OK, so the DIMMs I tried are all PN: SB 398708-061 These are PC2-5300 FBDIMMS. We had these on hand, I tried filling all 8 slots and the system wouldn't boot.
Dumitru, will you take care of ordering more compatible RAM?
Sheeri: how much memory do you think we need for it? I need a number, thanks :)
(In reply to Dustin J. Mitchell [:dustin] from comment #4) > Dumitru, will you take care of ordering more compatible RAM? Yes, I will, I just to know how much more.
Let's just max out the machine. 8 slot x 8GB.
The entire db is 57G right now, so getting 64 Gb of RAM is an excellent idea, and completely recommended even ignoring the fact that Zandr already suggested it.
per Releng/IT meeting today, installing this RAM will require a tree closure as well as coordinated downtime with all groups in mozilla who have production database instances on that machine. (In reply to Dumitru Gherman [:dumitru] from comment #6) > (In reply to Dustin J. Mitchell [:dustin] from comment #4) > > Dumitru, will you take care of ordering more compatible RAM? > > Yes, I will, I just to know how much more. dumitru: can you link the bug# which tracks purchasing RAM, so we can start coordinating with all the groups for this install?
Flags: needs-treeclosure?
> dumitru: can you link the bug# which tracks purchasing RAM, so we can start > coordinating with all the groups for this install? This bug.
(In reply to Dustin J. Mitchell [:dustin] from comment #10) > > dumitru: can you link the bug# which tracks purchasing RAM, so we can start > > coordinating with all the groups for this install? > > This bug. ah, ok. 1) is the ram ordered yet? 2) whats the ETA for arrival so we can start coordinating? (This particular work requires coordination with all groups that use the db server, not just RelEng.)
(In reply to John O'Duinn [:joduinn] from comment #11) > ah, ok. 1) is the ram ordered yet? 2) whats the ETA for arrival so we can > start coordinating? (This particular work requires coordination with all > groups that use the db server, not just RelEng.) Per comment #10, that information will be added to this bug when it becomes available.
Assignee: dgherman → phong
This server was originally purchased in 2006? Should we upgrade to new hardware?
The buildbot databases are moving to a new cluster in bug 707203, and I think this host is slated for replacement as part of the scl3 move. But in the mean time, more memory will help quite a bit.
It should arrive on Friday.
This should be in today. Catlee, as buildduty, how quickly can you schedule a downtime for this? Other groups are affected as well, so please let us know here or in IRC what time you'd like to shoot for so we can verify with them.
Per conversations in #ops, this is scheduled for tomorrow (Friday) morning. The window is 7-10am Pacific. Catlee will close trees and begin draining jobs at 7am. Richard will be available starting at 8am pacific, and will wait for a "go" from sheeri and catlee to take the host down and add the RAM. This will happen as soon as possible, and no later than 8:30. Catlee has gotten verbal sign-offs, and will begin announcing now. I don't have specific notification contact information for MDN and QA, who will also be affected by this downtime.
Flags: needs-treeclosure? → needs-treeclosure+
"Richard" => ":rbryce", I'm a dummy several times over. Sorry :(
Here's the downtime notice that went out: ---- IT has just received more RAM for the database server that's been giving us so much fun lately. They are ready to install it as soon as possible. This work is being tracked in https://bugzilla.mozilla.org/show_bug.cgi?id=713092. Given that: a) the db server is currently running very close to its upper memory limits b) is known to cause problems once the database server starts swapping c) load is likely to increase next week with people returning from vacation I'm asking for permission to have a downtime tomorrow morning. We'll close the trees at 5am PST to allow time for running jobs to finish up. The server will be shut off between 8:00 and 8:30am PST to install the new ram. Once the master is back online, the trees can be reopened in short order. We expect to have all trees reopened by 9am PST. Apologies for the short notice, and for the necessity of having this planned downtime so shortly after this morning's unplanned outage. Hopefully this maintenance will alleviate future issues, and doing this work tomorrow avoids needing to schedule a downtime during a historically busy first week after vacations. If there are any objections, please let us know immediately. Given the short notice, and the large number of people on PTO over the holidays, this time we are assuming "silence equals consent". ---- So the window agreed in comment 17 has changed to 5-9am. The plan within that is: 5am close trees, begin draining jobs 7am begin clean shutdown of MySQL (or later if this will not require 1h) 8am power off machine, swap RAM 8:30am debugging time 9am machine is back online and trees re-open I haven't heard a positive ack from :rbryce, so this may require slight revision.
HW Upgrade: Server was taken down @ 0800 pacific. 8GB of ram was upgraded to 64GB of ram. The server booted with no issues. Mysql Changes: (:sheeri) set tm-b01-master01 to have 16Gb innodb buffer pool, based on 11Gb of innodb indexes and data; 20Gb myisam key cache, based on 16G myisam indexes. The box was upgraded to 64 Gb RAM, the other 32G is for disk cache, which MySQL uses for MyISAM data.
I am closing this out.
Status: NEW → RESOLVED
Closed: 14 years ago
Resolution: --- → FIXED
Product: mozilla.org → Data & BI Services Team
You need to log in before you can comment on or make changes to this bug.