975331 - sumotools1.webapp.phx1.mozilla.com:MySQL Replication Slow is CRITICAL

Status: RESOLVED DUPLICATE of bug 978002

(Data & BI Services Team :: DB: MySQL, task)

Description

[Ashlee Nguyen [:ashlee]]

[00:36] < nagios-phx1> | Fri 00:36:44 PST [1473]
          sumotools1.webapp.phx1.mozilla.com:MySQL Replication Slow is CRITICAL:
          Replication Stopped - Last error:
          (http://m.mozilla.org/MySQL+Replication+Slow)

Comment 1

[Rick Bryce [:rbryce]]

this just alerted again.  

MOTD says this is a standalone master.  Is this check even valid?

Comment 2

[Ludovic Hirlimann [:Usul]]

(In reply to Rick Bryce [:rbryce] from comment #1)
> this just alerted again.  
> 
> MOTD says this is a standalone master.  Is this check even valid?

ditto.

Comment 3

[Sheeri Cabral [:sheeri]]

Yes, it is a slave of sumotools, which was made a while ago (it started out as a standalone master). I have changed puppet to reflect the fact that this is a slave and a master, but not really master/master - it's a slave that gets written to.

The reason this keeps getting critical is that the checksums on sumotools have differences, in 18 different tables (select * from percona.checksums where this_crc!=master_crc;)

This needs to be fixed, and then replication shouldn't break. (I can see in the error log that the problems are of a nature like:
2014-02-22 01:24:21 2358 [ERROR] Slave SQL: Error 'Duplicate entry '7786-7173-link' for key 'wiki_documentlink_linked_from_id_5f6e495fb11b7af7_uniq'' on query. Default database: 'support_mozilla_com'. Query: 'INSERT INTO `wiki_documentlink` (`linked_to_id`, `linked_from_id`, `kind`) VALUES (7173, 7786, 'link')', Error_code: 1062

This means the database already has that entry, and we need to fix sumotools so it has the correct data.)

Comment 4

[Sheeri Cabral [:sheeri]]

Brandon fixed this over in 978002...