913553 - Restart pg_dump backups of socorro's database

Status: RESOLVED FIXED

(Data & BI Services Team :: DB: MySQL, task)

Description

[Selena Deckelmann :selenamarie :selena]

I was investigating bug 818989, and wasn't able to find recent weekly dumps of Socorro's databases: 

[[e-mail]1 data]# find pgdumps/
pgdumps/
pgdumps/9.2
pgdumps/9.2/main
pgdumps/9.2/main/breakpad-db-globals-20130706.dump
pgdumps/9.2/main/breakpad-db-20130713.dump
pgdumps/9.2/main/breakpad-db-globals-20130713.dump
pgdumps/9.2/main/breakpad-db-20130706.dump
[[e-mail]1 data]# 


Could you let me know where the new location is? Thanks!

Comment 1

[Sheeri Cabral [:sheeri]]

We stopped the weekly backups in mid-July due to space issues. We're working on compressing the dumps so that we can store them again, but it's low priority.

We're still taking the full base backups daily and storing the last 3 days. Those are in /data/pgbasebackups. Bug 818989 refers only to the base backups, not the weekly dumps. Did you need a historical backup?

Comment 2

[Selena Deckelmann :selenamarie :selena]

(In reply to Sheeri Cabral [:sheeri] from comment #1)
> We stopped the weekly backups in mid-July due to space issues. We're working
> on compressing the dumps so that we can store them again, but it's low
> priority.
> 
> We're still taking the full base backups daily and storing the last 3 days.
> Those are in /data/pgbasebackups. Bug 818989 refers only to the base
> backups, not the weekly dumps. Did you need a historical backup?

I'm super concerned about not having dumps on a weekly basis because of the intermitant failures in the base backups. The dumps were our insurance.  Also, there's a lot of value in having dumps for partial failures, like memory or disk corruption in specific tables. 

Could we have one dump preserved each week? It seems like there's enough space for it currently.

Comment 3

[Sheeri Cabral [:sheeri]]

FWIW the base backup intermittent failures have been fixed, and have been for a long time.

But sure, we can have a dump preserved weekly; that might take a bit. It'd be much easier to store more of the base backups, compressed.

Comment 4

[Selena Deckelmann :selenamarie :selena]

(In reply to Sheeri Cabral [:sheeri] from comment #3)
> FWIW the base backup intermittent failures have been fixed, and have been
> for a long time.

Unfortunately, that's not true: https://bugzilla.mozilla.org/show_bug.cgi?id=818989#c12

> But sure, we can have a dump preserved weekly; that might take a bit. It'd
> be much easier to store more of the base backups, compressed.

I feel that the risk of data loss is very high, and I strongly argue that we always support weekly dumps for Postgres.

Comment 5

[Laura Thomson :laura]

Hey, I'm pretty worried about this.  If we don't have space, let's just order more.  What can I do to make this happen?

Comment 6

[Sheeri Cabral [:sheeri]]

As per comment 3, we can store more compressed base backups right now, but doing weekly dumps will take longer - by end of october.

Comment 7

[Matt Pressman [:mpressman]]

backup4.db.phx1 is now set up again to handle weekly pg_dumps. The next run is scheduled for 10/12

Comment 8

[Matt Pressman [:mpressman]]

The 10/12 run was successful.

Comment 9

[Sheeri Cabral [:sheeri]]

Confirming:

[[e-mail]1 main]# pwd
/data/pgdumps/9.2/main
[[e-mail]1 main]# ls -lrth
total 83G
-rw-r--r-- 1 root root    0 Sep 24 02:02 breakpad-db-20130924.dump
-rw-r--r-- 1 root root    0 Sep 28 02:00 breakpad-db-20130928.dump
-rw-r--r-- 1 root root    0 Oct  5 02:00 breakpad-db-20131005.dump
-rw-r--r-- 1 root root  83G Oct 12 06:11 breakpad-db-20131012.dump
-rw-r--r-- 1 root root 4.1K Oct 12 06:11 breakpad-db-globals-20131012.dump