1032408 - Add check_postgres_backups to puppet

Status: RESOLVED FIXED

(Data & BI Services Team :: DB: MySQL, task)

Description

[Matt Pressman [:mpressman]]

Add check to puppet

Comment 1

[Matt Pressman [:mpressman]]

The check is in puppet committed in revision 89748. Next step is to add the check to the service group

Comment 2

[Sheeri Cabral [:sheeri]]

Looks like the check is in modules/nrpe/files/plugins/check_postgres_backups and it's using the old-style "check that a file has all the right stuff in it."

However, on backup4 in phx, that file doesn't exist:
[[e-mail]1 ~]# cat /var/run/backup-status
cat: /var/run/backup-status: No such file or directory

In addition, we no longer check backups that way - we directly connect to the database instances and check raw and sql backups, in 2 separate checks.

Compare how these checks are run:
check_raw_backups_dd
check_sql_backups_dd

They rely on a file with all the instance names in them; I'm not sure if the postgres backups have an equivalent.

Comment 3

[Sheeri Cabral [:sheeri]]

phx backups live on backup4.db.phx1.mozilla.com:
/data/backups/INSTANCE/pgdumps/VERSION/main/DBNAME-db-DATE.dump

e.g.
/data/backups/generic/pgdumps/9.2/main/air_mozilla_org-db-20160611.dump

AND
/data/backups/INSTANCE/pgbasebackups/VERSION/DATE

e.g. 
/data/backups/generic/pgbasebackups/9.2/2016072003

Comment 4

[Sheeri Cabral [:sheeri]]

So, we need 2 checks - one for pgdumps and one for pgbasebackups. Created bugs 1288800 and 1288801 for those checks, for the MOC.

I have started the puppetization of backups - for right now, I have made /etc/sysconfig/postgres-backup-clusters, ensured the /data structure exists (e.g. /data/INSTANCE/pgdumps and /data/INSTANCE/pgbasebackups) and /usr/local/bin/backup_postgres.sh.

Will continue next week with the puppetization of backups, since that's also needed for the onboarding documentation.

Comment 5

[Sheeri Cabral [:sheeri]]

puppetized base backup script to /usr/local/bin, and the cron and variable scripts for the base backups in /etc/cron.d/ and /usr/local/bin. Need to see if they worked tomorrow, not sure they all will, some variables are hard-coded and I ran out of time to make sure they all are right.

Comment 6

[Sheeri Cabral [:sheeri]]

pgbasebackups are puppetized appropriately. Will check they worked automatically overnight.

Working on pgdump puppetization now.

Comment 7

[Sheeri Cabral [:sheeri]]

pgdump puppetized.

Comment 8

[Sheeri Cabral [:sheeri]]

adding :arr as a cc, as the pgdump puppetization necessitated puppetagain1 to be the master, as documented https://mana.mozilla.org/wiki/display/BIDW/How+to+onboard+a+new+Postgres+server#HowtoonboardanewPostgresserver-Backups