766261 - Please page us about db backups

Status: VERIFIED FIXED

(mozilla.org Graveyard :: Server Operations, task)

Description

[Sheeri Cabral [:sheeri]]

We don't actually get physically paged for the backups not running...for example:

[13:31:49] <nagios-phx1> [118] backup1.db.phx1.mozilla.com:Database Backups is CRITICAL: CRITICAL: dev1.db.phx1 is not replicating, sumo is not replicating, addons:marketplace_mozilla_org sql 1.33 days, addons:mysql sql 1.33 days, addons:percona sql 1.33 days, addons:pfs2_mozilla_org sql 1.33 days, addons:plugins_mozilla_org sql 1.33 days, dev1.db.phx1:addons_dev sql 1.33 days, dev1.db.phx1:addons_forums_allizom_org sql 1.33 days, dev1.db.phx1:

Please page us. backups are important! (there are 3 backup servers.....)

Comment 1

[Ashish Vijayaram [:ashish]]

The 3 backup serves are:
* backup1.db.phx1
* backup2.db.phx1
* backup1.db.scl3

This needed some cleanup. There were two services with the same check but with different names, hosts and options. I've consolidated them now. For clarity - "MySQL DB Backups" will page DBAs like all other DB checks. In addition, it will also alert if the check output differs within the same state [1].


[1] So first alert for "CRITICAL: dev1.db.phx1 is not replicating", second alert for "CRITICAL: dev1.db.phx1 is not replicating, addons:marketplace_mozilla_org sql 1.33 days" but not if the output remains the same. Excellent? :)

Comment 2

[Sheeri Cabral [:sheeri]]

Mucho excelente!

Not verified just yet, but I believe in your work .....

Comment 3

[Sheeri Cabral [:sheeri]]

Verified!