Restart pg_dump backups of socorro's database

RESOLVED FIXED

Status

Data & BI Services Team
DB: MySQL
--
critical
RESOLVED FIXED
4 years ago
3 years ago

People

(Reporter: selenamarie, Assigned: sheeri)

Tracking

Details

I was investigating bug 818989, and wasn't able to find recent weekly dumps of Socorro's databases: 

[root@backup4.db.phx1 data]# find pgdumps/
pgdumps/
pgdumps/9.2
pgdumps/9.2/main
pgdumps/9.2/main/breakpad-db-globals-20130706.dump
pgdumps/9.2/main/breakpad-db-20130713.dump
pgdumps/9.2/main/breakpad-db-globals-20130713.dump
pgdumps/9.2/main/breakpad-db-20130706.dump
[root@backup4.db.phx1 data]# 


Could you let me know where the new location is? Thanks!
(Assignee)

Comment 1

4 years ago
We stopped the weekly backups in mid-July due to space issues. We're working on compressing the dumps so that we can store them again, but it's low priority.

We're still taking the full base backups daily and storing the last 3 days. Those are in /data/pgbasebackups. Bug 818989 refers only to the base backups, not the weekly dumps. Did you need a historical backup?
(In reply to Sheeri Cabral [:sheeri] from comment #1)
> We stopped the weekly backups in mid-July due to space issues. We're working
> on compressing the dumps so that we can store them again, but it's low
> priority.
> 
> We're still taking the full base backups daily and storing the last 3 days.
> Those are in /data/pgbasebackups. Bug 818989 refers only to the base
> backups, not the weekly dumps. Did you need a historical backup?

I'm super concerned about not having dumps on a weekly basis because of the intermitant failures in the base backups. The dumps were our insurance.  Also, there's a lot of value in having dumps for partial failures, like memory or disk corruption in specific tables. 

Could we have one dump preserved each week? It seems like there's enough space for it currently.
(Assignee)

Comment 3

4 years ago
FWIW the base backup intermittent failures have been fixed, and have been for a long time.

But sure, we can have a dump preserved weekly; that might take a bit. It'd be much easier to store more of the base backups, compressed.
(In reply to Sheeri Cabral [:sheeri] from comment #3)
> FWIW the base backup intermittent failures have been fixed, and have been
> for a long time.

Unfortunately, that's not true: https://bugzilla.mozilla.org/show_bug.cgi?id=818989#c12

> But sure, we can have a dump preserved weekly; that might take a bit. It'd
> be much easier to store more of the base backups, compressed.

I feel that the risk of data loss is very high, and I strongly argue that we always support weekly dumps for Postgres.

Comment 5

4 years ago
Hey, I'm pretty worried about this.  If we don't have space, let's just order more.  What can I do to make this happen?
(Assignee)

Comment 6

4 years ago
As per comment 3, we can store more compressed base backups right now, but doing weekly dumps will take longer - by end of october.
(Assignee)

Updated

4 years ago
Assignee: server-ops-database → scabral
Summary: Location of weekly dumps of Socorro Postgres → Restart pg_dump backups of socorro's database
backup4.db.phx1 is now set up again to handle weekly pg_dumps. The next run is scheduled for 10/12
The 10/12 run was successful.
(Assignee)

Comment 9

4 years ago
Confirming:

[root@backup4.db.phx1 main]# pwd
/data/pgdumps/9.2/main
[root@backup4.db.phx1 main]# ls -lrth
total 83G
-rw-r--r-- 1 root root    0 Sep 24 02:02 breakpad-db-20130924.dump
-rw-r--r-- 1 root root    0 Sep 28 02:00 breakpad-db-20130928.dump
-rw-r--r-- 1 root root    0 Oct  5 02:00 breakpad-db-20131005.dump
-rw-r--r-- 1 root root  83G Oct 12 06:11 breakpad-db-20131012.dump
-rw-r--r-- 1 root root 4.1K Oct 12 06:11 breakpad-db-globals-20131012.dump
Status: NEW → RESOLVED
Last Resolved: 4 years ago
Resolution: --- → FIXED
Product: mozilla.org → Data & BI Services Team
You need to log in before you can comment on or make changes to this bug.