540170 - Need to integrate Vertica backups into Bakbone (corp backup tool)

Status: RESOLVED FIXED

(mozilla.org Graveyard :: Server Operations, task)

Description

[Daniel Einspanjer [:dre] [:deinspanjer]]

Currently, we stage the data from the three cm-metricsdb0[1-3] machines onto the attached drive on cm-metricsetl01 (mounted on /home/dbadmin).

We really need to reclaim the space on this drive to use it for the etl01 MySQL database.

Could we hook up a USB drive or something to either db01 or etl01 and migrate the Vertica backup staging over to it?

Comment 1

[Pedro Alves]

We don't need space on etl01 anymore - we have plenty

Comment 2

[Daniel Einspanjer [:dre] [:deinspanjer]]

This bug wasn't supposed to be closed.
The Vertica backups are slowly getting larger, and we are now at the point that Nagios is warning that this disk is almost full.
We need to have a better way to stage the Vertica data so it can be backed up.
Either extra disk(s)s that we can copy the snapshot data onto, or a more complex approach where the backup daemon runs a script that takes the snapshot, copies the files directly from the machines, and when done, releases the snapshot.

Comment 4

[matthew zeier [:mrz]]

USB drives aren't rack friendly.  Possible to do if it's short term but I don't think it's the best solution.  Also implies you don't care if that drive dies.

Does this need to be high performance?  How much space do you need?  Will iscsi work?

Comment 5

[Daniel Einspanjer [:dre] [:deinspanjer]]

Worst case if it dies and takes more than a day to replace is we don't have backups.
Doesn't need to be high performance, but it might need to copy a few hundred gig or so. On normal days the transfer is incremental.
2TB would be nice for room for growth.
iSCSI or maybe NFS might be fine, I could even mount it on the three machines and have them sync their own files to it.

Comment 6

[Daniel Einspanjer [:dre] [:deinspanjer]]

I wiped out a bunch of files on the disk just to get rid of the critical page.  The files will come back again tonight when it attempts to back up though, and it looks like we are now out of space cause it hit 0% last night.

Comment 7

[Daniel Einspanjer [:dre] [:deinspanjer]]

Need to get resolution on this.  I wiped out half a year of versioncheck data (that can be regenerated) in order to try to make it fit, but we are definitely at a critical stage here.

Comment 8

[Daniel Einspanjer [:dre] [:deinspanjer]]

Summary of the current backup process in case it helps for planning:

* at 00:00 PST, etl01 fires a cron job to begin the backup.
* cm-metricsetl01:/home/dbadmin/vertica_cluster_backup/backup.sh performs the following rough logic:
* clean up previous run scripts and archive run logs
* build new run scripts and execute them
* execute SQL command on Vertica cluster to create a snapshot point for a stable hot backup
* while that snapshot point is active (i.e. a current transaction)
* get a list of all data files that are part of this snapshot
* rsync all those files from each of the three servers onto the etl01 backup staging directory
* delete any old files that are no longer part of backup (just the --delete flag of rsync)
* backup config files
* when rsync is finished, close SQL transaction which closes the snapshot window

We could easily have the backups be taken from each machine individually.. the only requirement is that the snapshot SQL command be executed and that the transaction not get closed until all the files are rsynced/backed-up.

Comment 9

[Justin Fitzhugh]

To be clear, we don't need the space if we can:

* run a command before the backup job runs
* keep the command open while the backup is running
* end the command when the backup is done (Daniel can give details)

If you can do the above, we don't need the 2tb of space.  Aravind and Daniel - please sync up on this.

Comment 10

[Daniel Einspanjer [:dre] [:deinspanjer]]

Aravind, please ping me or call my extension if you want to discuss further.  I've gone through and added more comments to the backup.sh file mentioned in comment #8.

Comment 11

[Aravind Gottipati [:aravind]]

After talking to Daniel, here is what we will do.

1. Mount the three machines that have data to be backuped up as NFS mounts onto the central machine (cm-metricsetl01).

2. Daniel will give me a script to be called that will quiesce the system and generate a list of files that need to be backed up.  This script will exist after this file list has been generated (even though it might spawn off additional processes that keep the db in the quiesce mode.

3. Bakbone will read this generate file and backup this list of files in it.

4. After the backup is done, bakbone will call another script to release the db back into its normal mode.

We have not tested how inclusion lists work with bakbone, so will have to watch this system pretty closely for a few days once we put this into place.

@Daniel, please comment here once you side of things are ready (the nfs mounts, and the scripts).

Comment 12

[matthew zeier [:mrz]]

Punt back when you have your script.

Comment 13

[Daniel Einspanjer [:dre] [:deinspanjer]]

The backup script was reconfigured to store the files for each db machine on a new directory on that machine.  The following four directories should be backed up in order to have a complete backup set:

cm-metricsetl01/home/dbadmin/vertica_cluster_backup
cm-metricsdb01/home/dbadmin/vertica_cluster_backup
cm-metricsdb02/home/dbadmin/vertica_cluster_backup
cm-metricsdb03/home/dbadmin/vertica_cluster_backup

Comment 14

[Shyam Mani [:fox2mike]]

Aravind is our backup dude!

Comment 15

[Aravind Gottipati [:aravind]]

302 to Phong.

Comment 16

[Phong Tran [:phong]]

I will add these to tonight backups.

Comment 17

[Phong Tran [:phong]]

we have a policy set up for /home so those should be included.  let me know if you want everything else to be excluded and only back up /home/dbadmin/vertica_cluster_backup

Comment 18

[Daniel Einspanjer [:dre] [:deinspanjer]]

Yikes, definitely has to be just the /home/dbadmin/vertica_cluster_backup directory, otherwise you'd be taking two copies of all the data.

Comment 19

[Phong Tran [:phong]]

updated policy to only back up /home/dbadmin/vertica_cluster_backup