557102 - [AMO] Add the slow query log to logrotate and sync to cm-vpn01

Status: RESOLVED FIXED

(mozilla.org Graveyard :: Server Operations, task)

Description

[Wil Clouser [:clouserw]]

Dave gave us a dump of the slow query log from the AMO master db and it's full of good stuff that we can focus on and fix.  I wish we would have had it sooner so we could keep an eye on these things, which brings us too:

Please chop up the master and slave slow query logs for AMO preview and production and sync them to cm-vpn01.  Thanks.

Comment 1

[matthew zeier [:mrz]]

Is this the sort of stuff we could catch in stage?

Comment 2

[Wil Clouser [:clouserw]]

I don't think we can catch all of it in stage.  The queries get slow because they are waiting on other queries and that's caused by traffic.  Slow queries don't necessarily mean failed queries, they are just good indications of stuff we should look at.

Comment 3

[Jeff Balogh (:jbalogh)]

We're still having issues with this; seeing ongoing logs would be a big help.

Comment 4

[timellis]

The slow query log is 3GB. Initially I'll do a "tail -10000" of the log and send you those entries. If those aren't enough, let me know, and I'll make it bigger. I'm hoping to have this done in the next couple hours.

Comment 5

[timellis]

We have decided cm-vpn01 is too open to view these logs so instead have created a new mount on khan to achieve the same thing. Here is the file:

khan: /mnt/amo01-master01/tm-amo01-master01-mysql-slow.log.2010-04-20

There will be a new file every day. It is generated as a tail -50000 of the slow log. Please do tell me if that's too many lines to be useful, because it generates a 16MB file currently.

If you do not see a new logfile appearing daily, or khan is not an acceptable alternative host, please reopen this bug.