731239 - AMO databases no longer getting dumped?

Status: RESOLVED FIXED

(Data & BI Services Team :: DB: MySQL, task)

Description

[Wil Clouser [:clouserw]]

It looks like we're no longer getting full AMO database dumps on cm-webdev01-master01:

> [clouserw@cm-webdev01-master01 addons_mozilla_org]$ pwd
> /data/backup-drop/addons/mysql/addons_mozilla_org
> [clouserw@cm-webdev01-master01 addons_mozilla_org]$ ls -l
> total 12
> -rw-r--r-- 1 root root 49 Feb 26 02:10 update_counts.2012.02.26.sql.gz
> -rw-r--r-- 1 root root 49 Feb 27 02:10 update_counts.2012.02.27.sql.gz
> -rw-r--r-- 1 root root 49 Feb 28 02:10 update_counts.2012.02.28.sql.gz

Comment 1

[Sheeri Cabral [:sheeri]]

clouserw - the full amo dumps that were happening were coming from the old tm-amo database. We do have full amo dumps from the regular db, I will ensure they are being copied to cm-webdev01-master01 as well.

Comment 2

[Sheeri Cabral [:sheeri]]

odd, in /etc/cron.d on the backups server we have:

# copy to cm-webdev01-master01
0 2 * * * root /data/backups/bin/cm-webdev01-master01.mozilla.org.sh

So it should work at 2 am every day. I'm running the script manually and it seems to be working:

[[e-mail]1 bin]# ./cm-webdev01-master01.mozilla.org.sh 
Advisory: Dump file /data/backups/addons/sqldumps/addons_mozilla_org/addons_mozilla_org.2012.02.28.sql.gz already exists, just copying
Running scp /data/backups/addons/sqldumps/addons_mozilla_org/addons_mozilla_org.2012.02.28.sql.gz [e-mail]:/data/backup-drop/addons/mysql/addons_mozilla_org/
addons_mozilla_org.2012.02.28.sql.gz                                                                                                                  25% 1171MB   6.3MB/s   08:58 ETA

and it is indeed appearing on the right server:

[root@cm-webdev01-master01 addons_mozilla_org]# ls -lh
total 1.3G
-rwx------ 1 root root 1.3G Feb 28 09:14 addons_mozilla_org.2012.02.28.sql.gz
-rw-r--r-- 1 root root   49 Feb 26 02:10 update_counts.2012.02.26.sql.gz
-rw-r--r-- 1 root root   49 Feb 27 02:10 update_counts.2012.02.27.sql.gz
-rw-r--r-- 1 root root   49 Feb 28 02:10 update_counts.2012.02.28.sql.gz
[root@cm-webdev01-master01 addons_mozilla_org]# 

I'll update this ticket when the script is finished, and I'll check for scripts that might be deleting the files. Also, I'll double-check the update_counts scripts, as those look wrongly small.

Comment 3

[Sheeri Cabral [:sheeri]]

OK, I found a problem with the update_counts table, and I think the 2 scripts were conflicting with each other. I've set them to run an hour apart, and fixed the update_counts one.

[root@cm-webdev01-master01 addons_mozilla_org]# ls -lh
total 4.5G
-rwx------ 1 root root 4.5G Feb 28 11:52 addons_mozilla_org.2012.02.28.sql.gz
-rw-r--r-- 1 root root   49 Feb 26 02:10 update_counts.2012.02.26.sql.gz
-rw-r--r-- 1 root root   49 Feb 27 02:10 update_counts.2012.02.27.sql.gz
-rw-r--r-- 1 root root 3.5M Feb 28 12:09 update_counts.2012.02.28.sql.gz

Is that more like what you are expecting?

Comment 4

[Wil Clouser [:clouserw]]

that looks good

Comment 5

[Sheeri Cabral [:sheeri]]

The update_counts exports are working again, but for some reason the full backups of addons are not being copied properly.

[root@cm-webdev01-master01 addons_mozilla_org]# ls -l
total 4660808
-rwx------ 1 root root 4760422639 Feb 28 11:52 addons_mozilla_org.2012.02.28.sql.gz
-rw-r--r-- 1 root root         49 Feb 27 02:10 update_counts.2012.02.27.sql.gz
-rw-r--r-- 1 root root    3623454 Feb 28 12:09 update_counts.2012.02.28.sql.gz
-rw-r--r-- 1 root root    3941614 Feb 29 03:00 update_counts.2012.02.29.sql.gz

I'll run the script manually to check and try to figure out why this is still having problems.

Comment 6

[Sheeri Cabral [:sheeri]]

W00t! Finally the timing is right:

[root@cm-webdev01-master01 addons_mozilla_org]# ls -l
total 13985916
-rwx------ 1 root root 4760422639 Feb 28 11:52 addons_mozilla_org.2012.02.28.sql.gz
-rwx------ 1 root root 4764805785 Feb 29 12:41 addons_mozilla_org.2012.02.29.sql.gz
-rwx------ 1 root root 4770812774 Mar  1 04:05 addons_mozilla_org.2012.03.01.sql.gz
-rw-r--r-- 1 root root    3623454 Feb 28 12:09 update_counts.2012.02.28.sql.gz
-rw-r--r-- 1 root root    3941614 Feb 29 03:00 update_counts.2012.02.29.sql.gz
-rw-r--r-- 1 root root    3944196 Mar  1 02:00 update_counts.2012.03.01.sql.gz

Note the 4:05 am copy time of today's backup of addons.

Also note:
[root@cm-webdev01-master01 addons_mozilla_org]# more /etc/cron.d/prune_backups 
0 5 * * * root /usr/bin/find /data/backup-drop/ -type f -mtime +2 -exec rm -f {} \;

The backup from the 28th hasn't been pruned yet because it was only 1.5 days old when the prune script ran. The point is, we don't have to worry about the partition filling up.

Resolving!

Comment 7

[Wil Clouser [:clouserw]]

thanks!

Comment 8

[Sheeri Cabral [:sheeri]]

FYI this is happening again, as far as I can tell, due to a netflow issue, so I've opened up bug 779820 for it.

Comment 9

[Sheeri Cabral [:sheeri]]

Confirmed that the netflow is now open and backup copies are working again:

[root@webdev1 addons_mozilla_org]# ls -rtlh
total 22G
-rwx------ 1 root root 5.5G Aug 10 12:18 addons_mozilla_org.2012.08.10.sql.gz
-rw-r--r-- 1 root root 3.6M Aug 11 02:00 update_counts.2012.08.11.sql.gz
-rwx------ 1 root root 5.5G Aug 11 04:03 addons_mozilla_org.2012.08.11.sql.gz
-rw-r--r-- 1 root root 3.3M Aug 12 02:00 update_counts.2012.08.12.sql.gz
-rwx------ 1 root root 5.5G Aug 12 04:03 addons_mozilla_org.2012.08.12.sql.gz
-rw-r--r-- 1 root root 3.3M Aug 13 02:00 update_counts.2012.08.13.sql.gz
-rwx------ 1 root root 5.5G Aug 13 04:24 addons_mozilla_org.2012.08.13.sql.gz