848154 - backup2.db.phx1.mozilla.com:MySQL DB Backups is CRITICAL: CRITICAL: generic has a replication error

Status: RESOLVED FIXED

(Data & BI Services Team :: DB: MySQL, task)

Description

[Matt Pressman [:mpressman]]

14:58:32] <nagios-phx1> Tue 14:58:32 PST [123] backup2.db.phx1.mozilla.com:MySQL DB Backups is CRITICAL: CRITICAL: generic has a replication error.

Comment 1

[Matt Pressman [:mpressman]]

mysql> show slave status\G
*************************** 1. row ***************************
               Slave_IO_State: Waiting for master to send event
                  Master_Host: generic1.db.phx1.mozilla.com
                  Master_User: repl
                  Master_Port: 3306
                Connect_Retry: 60
              Master_Log_File: generic1-bin.000814
          Read_Master_Log_Pos: 341934579
               Relay_Log_File: generic-phx1-relay-bin.003252
                Relay_Log_Pos: 185887631
        Relay_Master_Log_File: generic1-bin.000800
             Slave_IO_Running: Yes
            Slave_SQL_Running: No
              Replicate_Do_DB:
          Replicate_Ignore_DB:
           Replicate_Do_Table:
       Replicate_Ignore_Table:
      Replicate_Wild_Do_Table:
  Replicate_Wild_Ignore_Table:
                   Last_Errno: 1146
                   Last_Error: Error 'Table 'wiki_mozilla_org.t0' doesn't exist' on query. Default database: 'wiki_mozilla_org'. Query: 'INSERT /* SMW::executeQueries 10.8.81.215 */ IGNORE INTO `t0` SELECT t8.s_id FROM `smw_atts2` AS t8  WHERE (t8.value_xsd='Security') AND t8.p_id='129''
                 Skip_Counter: 0
          Exec_Master_Log_Pos: 733866284
              Relay_Log_Space: 13080838081
              Until_Condition: None
               Until_Log_File:
                Until_Log_Pos: 0
           Master_SSL_Allowed: No
           Master_SSL_CA_File:
           Master_SSL_CA_Path:
              Master_SSL_Cert:
            Master_SSL_Cipher:
               Master_SSL_Key:
        Seconds_Behind_Master: NULL
Master_SSL_Verify_Server_Cert: No
                Last_IO_Errno: 0
                Last_IO_Error:
               Last_SQL_Errno: 1146
               Last_SQL_Error: Error 'Table 'wiki_mozilla_org.t0' doesn't exist' on query. Default database: 'wiki_mozilla_org'. Query: 'INSERT /* SMW::executeQueries 10.8.81.215 */ IGNORE INTO `t0` SELECT t8.s_id FROM `smw_atts2` AS t8  WHERE (t8.value_xsd='Security') AND t8.p_id='129''
1 row in set (0.00 sec)

Comment 2

[Matt Pressman [:mpressman]]

t0 is a temporary table and slave is about a week behind. I looked into the binlogs and discovered they are used to perform selects, no dml.

Depending on creation time and slave status, I wanted to make sure these weren't needed, but something to keep an eye on for data consistency.

I skipped the statement and it moved on.

mysql> set global sql_slave_skip_counter=1; start slave;
Query OK, 0 rows affected (0.00 sec)

Query OK, 0 rows affected (0.00 sec)

mysql> show slave status\G
*************************** 1. row ***************************
               Slave_IO_State: Waiting for master to send event
                  Master_Host: generic1.db.phx1.mozilla.com
                  Master_User: repl
                  Master_Port: 3306
                Connect_Retry: 60
              Master_Log_File: generic1-bin.000814
          Read_Master_Log_Pos: 343065523
               Relay_Log_File: generic-phx1-relay-bin.003254
                Relay_Log_Pos: 804271
        Relay_Master_Log_File: generic1-bin.000800
             Slave_IO_Running: Yes
            Slave_SQL_Running: Yes
              Replicate_Do_DB:
          Replicate_Ignore_DB:
           Replicate_Do_Table:
       Replicate_Ignore_Table:
      Replicate_Wild_Do_Table:
  Replicate_Wild_Ignore_Table:
                   Last_Errno: 0
                   Last_Error:
                 Skip_Counter: 0
          Exec_Master_Log_Pos: 734691336
              Relay_Log_Space: 12896060178
              Until_Condition: None
               Until_Log_File:
                Until_Log_Pos: 0
           Master_SSL_Allowed: No
           Master_SSL_CA_File:
           Master_SSL_CA_Path:
              Master_SSL_Cert:
            Master_SSL_Cipher:
               Master_SSL_Key:
        Seconds_Behind_Master: 236861
Master_SSL_Verify_Server_Cert: No
                Last_IO_Errno: 0
                Last_IO_Error:
               Last_SQL_Errno: 0
               Last_SQL_Error:
1 row in set (0.00 sec)