765939 - Need backup disk/host for socorro in different data center than phx1

Status: RESOLVED FIXED

(Infrastructure & Operations :: Storage, task)

Description

[Matt Pressman [:mpressman]]

We will need about 1TB of disk for socorro in a data center other than phx1 which is where the main prod servers as well as the dev machines are stored.

Comment 1

[Dan Parsons [:lerxst]]

Please specify exactly what kind of host you need to connect to this storage. We prefer not to give VMs with such huge disk space allocations, so a physical server might be better depending on what your needs are.

Comment 2

[Matt Pressman [:mpressman]]

There isn't a specific requirement per se. An iscsi/nfs what have you provided that we have offsite backup for our production crash data

Comment 3

[Dan Parsons [:lerxst]]

What machine will mount the storage that you're requesting?

Comment 4

[Dumitru Gherman [:dumitru]]

Short solution is to dump the data on tape and store it off-site.
Long term is our DR plan between datacenters.

So, for now, let me know where you dump the database so I can ingest it into tape.

Comment 5

[Matt Pressman [:mpressman]]

I'll be pushing a binary copy to socorro1.stage.db.phx1. Although currently the copy isn't compressed, I'll compress it in /pgdata. So the file name and location will be /pgdata/socorro-YYYYMonDD

The date matches the current convention for other db dumps. This will be done weekly.

Comment 6

[Matt Pressman [:mpressman]]

The full data set compressed nicely. All the way down to 128GB from 628GB.

Right now it is located on socorro1.stage.db.phx1.mozilla.com at
/pgdata/9.0/data.tar.gz

If you'd like I can move it to another directory on /pgdata and I can rename it, but all we care about is the last weeks worth of data so there is no need to continuously keep older archives.

I've put this into a job that runs every Sunday night.

Comment 7

[Dumitru Gherman [:dumitru]]

Per our discussion on IRC:
- backup folder should be "/pgdata/backups"
- we will use NetVault triggers to start the backup job after the pgdump finishes. Add this at the end of your script:

/usr/netvault/bin/nvtrigger -server backup1_ops_phx1_mozilla_com pgdata-backups

I tested it manually and it works.