1315341 - Automate the clean up Tableau daily backup files

Status: RESOLVED FIXED

(Infrastructure & Operations :: MOC: Service Requests, task)

Description

[[:josephine]]

This bug is to track the work to automate the clean up Tableau daily backup files.

Requirements will be specified.

Comment 1

[Ryan C [:ryanc] (UTC-4)]

(In reply to Josephine from comment #0)
> This bug is to track the work to automate the clean up Tableau daily backup
> files.
> 
> Requirements will be specified.

Awaiting requirements (directories, files, etc that need to be purged daily).

Comment 2

[Peter Radcliffe [:pir]]

Location and other details can be found in bug 1314771:

"Currently there is a daily backup that runs in tableau server and the file is put in C:\TableauBackups.

At the end of the month, the daily files are so large and it's hitting the disk space available in the server. I have a monthly manual routine to remove all daily backup files and leave only one backup file for every month. Please let me know if there is a way to do this better."


I believe, now that bacula backups are configured, the idea is to delete all but the last few days of files except one, per month.

The result should be one file for every previous month plus the last 5 days.

Comment 3

[[:josephine]]

Thanks Peter, that sounds right.

At the end of the month, I usually leave only 1 backup file, which is the last day of the month. So leaving one per month plus the last 5 days is good.

rchilds, please let me know if you need anything else.

Comment 4

[Ryan C [:ryanc] (UTC-4)]

That's everything I need to know - Rolling with that

Comment 5

[Ryan C [:ryanc] (UTC-4)]

Attachment: tableau_backup_cleanup1

Here's an early draft with some sample output to test,


PS C:\Users\serverops-admin\Downloads> .\listfiledates1.ps1
10/31/2015 tableau2-scl3-backup-2015-10-31-23-00-08.tsbak
10/31/2015
10/31/2015 10/31/2015
Dates match
---
11/30/2015 tableau2-scl3-backup-2015-11-30-23-00-08.tsbak
11/30/2015
11/30/2015 11/30/2015
Dates match
---
12/31/2015 tableau2-scl3-backup-2015-12-31-23-00-08.tsbak
12/31/2015
12/31/2015 12/31/2015
Dates match
---
1/31/2016 tableau2-scl3-backup-2016-01-31-23-00-09.tsbak
1/31/2016
1/31/2016 1/31/2016
Dates match
---
2/29/2016 tableau2-scl3-backup-2016-02-29-23-00-07.tsbak
2/29/2016
2/29/2016 2/29/2016
Dates match
---
3/31/2016 tableau2-scl3-backup-2016-03-31-23-00-06.tsbak
3/31/2016
3/31/2016 3/31/2016
Dates match
---
4/30/2016 tableau2-scl3-backup-2016-04-30-23-00-08.tsbak
4/30/2016
4/30/2016 4/30/2016
Dates match
---
5/31/2016 tableau2-scl3-backup-2016-05-31-23-00-08.tsbak
5/31/2016
5/31/2016 5/31/2016
Dates match
---
6/30/2016 tableau2-scl3-backup-2016-06-30-23-00-07.tsbak
6/30/2016
6/30/2016 6/30/2016
Dates match
---
7/31/2016 tableau2-scl3-backup-2016-07-31-23-00-07.tsbak
7/31/2016
7/31/2016 7/31/2016
Dates match
---
8/31/2016 tableau2-scl3-backup-2016-08-31-23-00-07.tsbak
8/31/2016
8/31/2016 8/31/2016
Dates match
---
10/1/2016 tableau2-scl3-backup-2016-09-30-23-55-07.tsbak
10/31/2016
10/31/2016 10/1/2016
Dates don't match
---
10/31/2016 tableau2-scl3-backup-2016-10-30-23-55-07.tsbak
10/31/2016
10/31/2016 10/31/2016
Dates match
---
11/1/2016 tableau2-scl3-backup-2016-10-31-23-55-06.tsbak
11/30/2016
11/30/2016 11/1/2016
Dates don't match
---
11/6/2016 tableau2-scl3-backup-2016-11-05-23-55-06.tsbak
11/30/2016
11/30/2016 11/6/2016
Dates don't match
---
11/7/2016 tableau2-scl3-backup-2016-11-06-23-55-06.tsbak
11/30/2016
11/30/2016 11/7/2016
Dates don't match
---

Comment 6

[[:josephine]]

Ryanc, please let me know if you need me to action on anything from your test result above.

Comment 7

[Ryan C [:ryanc] (UTC-4)]

(In reply to Josephine from comment #6)
> Ryanc, please let me know if you need me to action on anything from your
> test result above.

This needs just a bit more work and should get back around to this next week -- I will certainly let you know.

Comment 8

[[:josephine]]

Hi Ryan,

I'd like to remove some of the backup files because it's already eating up the disk space. Hope it's not affecting your test on your script. I do need to remove them and I'll do it EOD today.

Comment 9

[Ryan C [:ryanc] (UTC-4)]

That's fine

Comment 10

[Ryan C [:ryanc] (UTC-4)]

Attachment: tableau_backup_cleanup2

I initially wanted to do this in powershell, but looking at these systems it appears that they have python installed, so I went that route with this latest revision. Will polish and wrap this up next week.


tableau2-scl3-backup-2015-10-31-23-00-08.tsbak Monthly backup
tableau2-scl3-backup-2015-11-30-23-00-08.tsbak Monthly backup
tableau2-scl3-backup-2015-12-31-23-00-08.tsbak Monthly backup
tableau2-scl3-backup-2016-01-31-23-00-09.tsbak Monthly backup
tableau2-scl3-backup-2016-02-29-23-00-07.tsbak Monthly backup
tableau2-scl3-backup-2016-03-31-23-00-06.tsbak Monthly backup
tableau2-scl3-backup-2016-04-30-23-00-08.tsbak Monthly backup
tableau2-scl3-backup-2016-05-31-23-00-08.tsbak Monthly backup
tableau2-scl3-backup-2016-06-30-23-00-07.tsbak Monthly backup
tableau2-scl3-backup-2016-07-31-23-00-07.tsbak Monthly backup
tableau2-scl3-backup-2016-08-31-23-00-07.tsbak Monthly backup
tableau2-scl3-backup-2016-09-30-23-55-07.tsbak Monthly backup
tableau2-scl3-backup-2016-10-31-23-55-06.tsbak Monthly backup
tableau2-scl3-backup-2016-11-20-23-55-03.tsbak 4 days old
tableau2-scl3-backup-2016-11-21-23-55-03.tsbak 3 days old
tableau2-scl3-backup-2016-11-22-23-55-09.tsbak 2 days old
tableau2-scl3-backup-latest.tsbak 2 days old

['tableau2-scl3-backup-2016-11-15-23-55-07.tsbak', 'tableau2-scl3-backup-2016-11-16-23-55-03.tsbak', 'tableau2-scl3-backup-2016-11-17-23-55-08.tsbak', 'tableau2-scl3-backup-2016-11-18-23-55-03.tsbak', 'tableau2-scl3-backup-2016-11-19-23-55-03.tsbak']

Comment 11

[[:josephine]]

Great news!

Comment 12

[Ryan C [:ryanc] (UTC-4)]

(In reply to Josephine from comment #11)
> Great news!

Indeed -- Does there need to be any kind of log for when this runs?

Here's the script's current behavior in output, it'd just need to be changed to actually delete the backups. I'll include that final revision in my next comment,


tableau2-scl3-backup-2015-10-31-23-00-08.tsbak Monthly backup
tableau2-scl3-backup-2015-11-30-23-00-08.tsbak Monthly backup
tableau2-scl3-backup-2015-12-31-23-00-08.tsbak Monthly backup
tableau2-scl3-backup-2016-01-31-23-00-09.tsbak Monthly backup
tableau2-scl3-backup-2016-02-29-23-00-07.tsbak Monthly backup
tableau2-scl3-backup-2016-03-31-23-00-06.tsbak Monthly backup
tableau2-scl3-backup-2016-04-30-23-00-08.tsbak Monthly backup
tableau2-scl3-backup-2016-05-31-23-00-08.tsbak Monthly backup
tableau2-scl3-backup-2016-06-30-23-00-07.tsbak Monthly backup
tableau2-scl3-backup-2016-07-31-23-00-07.tsbak Monthly backup
tableau2-scl3-backup-2016-08-31-23-00-07.tsbak Monthly backup
tableau2-scl3-backup-2016-09-30-23-55-07.tsbak Monthly backup
tableau2-scl3-backup-2016-10-31-23-55-06.tsbak Monthly backup
tableau2-scl3-backup-2016-11-26-23-55-03.tsbak 5 days old
tableau2-scl3-backup-2016-11-27-23-55-03.tsbak 4 days old
tableau2-scl3-backup-2016-11-28-23-55-10.tsbak 3 days old
tableau2-scl3-backup-2016-11-29-23-55-03.tsbak 2 days old
tableau2-scl3-backup-2016-11-30-23-55-04.tsbak Monthly backup
---
backups to be deleted
tableau2-scl3-backup-2016-11-15-23-55-07.tsbak
tableau2-scl3-backup-2016-11-16-23-55-03.tsbak
tableau2-scl3-backup-2016-11-17-23-55-08.tsbak
tableau2-scl3-backup-2016-11-18-23-55-03.tsbak
tableau2-scl3-backup-2016-11-19-23-55-03.tsbak
tableau2-scl3-backup-2016-11-20-23-55-03.tsbak
tableau2-scl3-backup-2016-11-21-23-55-03.tsbak
tableau2-scl3-backup-2016-11-22-23-55-09.tsbak
tableau2-scl3-backup-2016-11-23-23-55-05.tsbak
tableau2-scl3-backup-2016-11-24-23-56-03.tsbak
tableau2-scl3-backup-2016-11-25-23-55-03.tsbak

Comment 13

[[:josephine]]

I think adding a log would be good in case there is an issue we do not anticipate.

Is there a way to know if bacula's backup successful or fail? The reason I asked is I would like the delete script you created to not run when bacula backup fails.

Also, we're moving tableau servers to VM. I would like to move this script to the new prod server (tableau6) when this is completed.

Will send an email out to MOC on the details of the tableau VMs.

Thanks,
Josephine

Comment 14

[Peter Radcliffe [:pir]]

The reason I suggested keeping the last 5 days was just in case bacula fails for a day or two, it gives time for those files to get backed up.

:rtucker I assume you'd know if bacula fails on these hosts? Is there a notification or a way to tell on the host itself?

Comment 15

[Rob Tucker [:rtucker]]

5 days is fine as well. I can configure bacula to send an email to whomever should backups fail.

Comment 16

[Ryan C [:ryanc] (UTC-4)]

Attachment: tableau_backup_cleanup3

Alright,

Here's the third revision with logging, I'll get this reviewed in Hawaii and get it out for use soon after.

Comment 17

[Ryan C [:ryanc] (UTC-4)]

Comment on attachment 8816732 [details]
tableau_backup_cleanup3

Keegan,

Let me know your thoughts on this script

Comment 18

[Keegan Ferrando [:fauweh]]

Comment on attachment 8816732 [details]
tableau_backup_cleanup3

as per conversation, please manage the log file properly (rotate, etc)

Comment 19

[Ryan C [:ryanc] (UTC-4)]

Attachment: tableau_backup_cleanup4

Updated script to truncate log if it's over 1Mb.

Josephine gave me the go-ahead to test the script yesterday and it ran with expected behavior,

2016-12-09 19:26:28.024000
tableau2-scl3-backup-2015-10-31-23-00-08.tsbak monthly backup
tableau2-scl3-backup-2015-11-30-23-00-08.tsbak monthly backup
tableau2-scl3-backup-2015-12-31-23-00-08.tsbak monthly backup
tableau2-scl3-backup-2016-01-31-23-00-09.tsbak monthly backup
tableau2-scl3-backup-2016-02-29-23-00-07.tsbak monthly backup
tableau2-scl3-backup-2016-03-31-23-00-06.tsbak monthly backup
tableau2-scl3-backup-2016-04-30-23-00-08.tsbak monthly backup
tableau2-scl3-backup-2016-05-31-23-00-08.tsbak monthly backup
tableau2-scl3-backup-2016-06-30-23-00-07.tsbak monthly backup
tableau2-scl3-backup-2016-07-31-23-00-07.tsbak monthly backup
tableau2-scl3-backup-2016-08-31-23-00-07.tsbak monthly backup
tableau2-scl3-backup-2016-09-30-23-55-07.tsbak monthly backup
tableau2-scl3-backup-2016-10-31-23-55-06.tsbak monthly backup
tableau2-scl3-backup-2016-11-30-23-55-04.tsbak monthly backup
tableau2-scl3-backup-2016-12-04-23-55-06.tsbak 5 days old
tableau2-scl3-backup-2016-12-05-23-55-05.tsbak 4 days old
tableau2-scl3-backup-2016-12-06-23-55-04.tsbak 3 days old
tableau2-scl3-backup-2016-12-07-23-55-04.tsbak 2 days old
tableau2-scl3-backup-2016-12-08-23-55-05.tsbak 1 days old
---
backups to be deleted
tableau2-scl3-backup-2016-11-15-23-55-07.tsbak
tableau2-scl3-backup-2016-11-16-23-55-03.tsbak
tableau2-scl3-backup-2016-11-17-23-55-08.tsbak
tableau2-scl3-backup-2016-11-18-23-55-03.tsbak
tableau2-scl3-backup-2016-11-19-23-55-03.tsbak
tableau2-scl3-backup-2016-11-20-23-55-03.tsbak
tableau2-scl3-backup-2016-11-21-23-55-03.tsbak
tableau2-scl3-backup-2016-11-22-23-55-09.tsbak
tableau2-scl3-backup-2016-11-23-23-55-05.tsbak
tableau2-scl3-backup-2016-11-24-23-56-03.tsbak
tableau2-scl3-backup-2016-11-25-23-55-03.tsbak
tableau2-scl3-backup-2016-11-26-23-55-03.tsbak
tableau2-scl3-backup-2016-11-27-23-55-03.tsbak
tableau2-scl3-backup-2016-11-28-23-55-10.tsbak
tableau2-scl3-backup-2016-11-29-23-55-03.tsbak
tableau2-scl3-backup-2016-12-01-23-55-03.tsbak
tableau2-scl3-backup-2016-12-02-23-55-03.tsbak
tableau2-scl3-backup-2016-12-03-23-55-07.tsbak


2016-12-09 19:32:48.048000
tableau2-scl3-backup-2015-10-31-23-00-08.tsbak monthly backup
tableau2-scl3-backup-2015-11-30-23-00-08.tsbak monthly backup
tableau2-scl3-backup-2015-12-31-23-00-08.tsbak monthly backup
tableau2-scl3-backup-2016-01-31-23-00-09.tsbak monthly backup
tableau2-scl3-backup-2016-02-29-23-00-07.tsbak monthly backup
tableau2-scl3-backup-2016-03-31-23-00-06.tsbak monthly backup
tableau2-scl3-backup-2016-04-30-23-00-08.tsbak monthly backup
tableau2-scl3-backup-2016-05-31-23-00-08.tsbak monthly backup
tableau2-scl3-backup-2016-06-30-23-00-07.tsbak monthly backup
tableau2-scl3-backup-2016-07-31-23-00-07.tsbak monthly backup
tableau2-scl3-backup-2016-08-31-23-00-07.tsbak monthly backup
tableau2-scl3-backup-2016-09-30-23-55-07.tsbak monthly backup
tableau2-scl3-backup-2016-10-31-23-55-06.tsbak monthly backup
tableau2-scl3-backup-2016-11-30-23-55-04.tsbak monthly backup
tableau2-scl3-backup-2016-12-04-23-55-06.tsbak 5 days old
tableau2-scl3-backup-2016-12-05-23-55-05.tsbak 4 days old
tableau2-scl3-backup-2016-12-06-23-55-04.tsbak 3 days old
tableau2-scl3-backup-2016-12-07-23-55-04.tsbak 2 days old
tableau2-scl3-backup-2016-12-08-23-55-05.tsbak 1 days old
---
backups to be deleted


Moving forward this script should be ran daily at 1AM

Comment 20

[Ryan C [:ryanc] (UTC-4)]

Calling this good - Feel free to reopen if anything further is needed

Ping me on IRC if you need assistance placing this script