Please page us about db backups

VERIFIED FIXED

Status

mozilla.org Graveyard
Server Operations
VERIFIED FIXED
6 years ago
3 years ago

People

(Reporter: sheeri, Assigned: ashish)

Tracking

Details

(Reporter)

Description

6 years ago
We don't actually get physically paged for the backups not running...for example:

[13:31:49] <nagios-phx1> [118] backup1.db.phx1.mozilla.com:Database Backups is CRITICAL: CRITICAL: dev1.db.phx1 is not replicating, sumo is not replicating, addons:marketplace_mozilla_org sql 1.33 days, addons:mysql sql 1.33 days, addons:percona sql 1.33 days, addons:pfs2_mozilla_org sql 1.33 days, addons:plugins_mozilla_org sql 1.33 days, dev1.db.phx1:addons_dev sql 1.33 days, dev1.db.phx1:addons_forums_allizom_org sql 1.33 days, dev1.db.phx1:

Please page us. backups are important! (there are 3 backup servers.....)
(Assignee)

Comment 1

6 years ago
The 3 backup serves are:
* backup1.db.phx1
* backup2.db.phx1
* backup1.db.scl3

This needed some cleanup. There were two services with the same check but with different names, hosts and options. I've consolidated them now. For clarity - "MySQL DB Backups" will page DBAs like all other DB checks. In addition, it will also alert if the check output differs within the same state [1].


[1] So first alert for "CRITICAL: dev1.db.phx1 is not replicating", second alert for "CRITICAL: dev1.db.phx1 is not replicating, addons:marketplace_mozilla_org sql 1.33 days" but not if the output remains the same. Excellent? :)
Assignee: server-ops → ashish
Status: NEW → RESOLVED
Last Resolved: 6 years ago
Resolution: --- → FIXED
(Reporter)

Comment 2

6 years ago
Mucho excelente!

Not verified just yet, but I believe in your work .....
(Reporter)

Comment 3

6 years ago
Verified!
Status: RESOLVED → VERIFIED
Product: mozilla.org → mozilla.org Graveyard
You need to log in before you can comment on or make changes to this bug.