AMO databases no longer getting dumped?

RESOLVED FIXED

Status

Data & BI Services Team
DB: MySQL
RESOLVED FIXED
6 years ago
4 years ago

People

(Reporter: clouserw, Assigned: sheeri)

Tracking

Details

(Reporter)

Description

6 years ago
It looks like we're no longer getting full AMO database dumps on cm-webdev01-master01:

> [clouserw@cm-webdev01-master01 addons_mozilla_org]$ pwd
> /data/backup-drop/addons/mysql/addons_mozilla_org
> [clouserw@cm-webdev01-master01 addons_mozilla_org]$ ls -l
> total 12
> -rw-r--r-- 1 root root 49 Feb 26 02:10 update_counts.2012.02.26.sql.gz
> -rw-r--r-- 1 root root 49 Feb 27 02:10 update_counts.2012.02.27.sql.gz
> -rw-r--r-- 1 root root 49 Feb 28 02:10 update_counts.2012.02.28.sql.gz
Assignee: server-ops → server-ops-database
Component: Server Operations → Server Operations: Database
QA Contact: phong → cshields
(Assignee)

Comment 1

6 years ago
clouserw - the full amo dumps that were happening were coming from the old tm-amo database. We do have full amo dumps from the regular db, I will ensure they are being copied to cm-webdev01-master01 as well.
Assignee: server-ops-database → scabral
Status: NEW → ASSIGNED
(Assignee)

Comment 2

6 years ago
odd, in /etc/cron.d on the backups server we have:

# copy to cm-webdev01-master01
0 2 * * * root /data/backups/bin/cm-webdev01-master01.mozilla.org.sh

So it should work at 2 am every day. I'm running the script manually and it seems to be working:

[root@db-backup1.ops.phx1 bin]# ./cm-webdev01-master01.mozilla.org.sh 
Advisory: Dump file /data/backups/addons/sqldumps/addons_mozilla_org/addons_mozilla_org.2012.02.28.sql.gz already exists, just copying
Running scp /data/backups/addons/sqldumps/addons_mozilla_org/addons_mozilla_org.2012.02.28.sql.gz root@cm-webdev01-master01.mozilla.org:/data/backup-drop/addons/mysql/addons_mozilla_org/
addons_mozilla_org.2012.02.28.sql.gz                                                                                                                  25% 1171MB   6.3MB/s   08:58 ETA

and it is indeed appearing on the right server:

[root@cm-webdev01-master01 addons_mozilla_org]# ls -lh
total 1.3G
-rwx------ 1 root root 1.3G Feb 28 09:14 addons_mozilla_org.2012.02.28.sql.gz
-rw-r--r-- 1 root root   49 Feb 26 02:10 update_counts.2012.02.26.sql.gz
-rw-r--r-- 1 root root   49 Feb 27 02:10 update_counts.2012.02.27.sql.gz
-rw-r--r-- 1 root root   49 Feb 28 02:10 update_counts.2012.02.28.sql.gz
[root@cm-webdev01-master01 addons_mozilla_org]# 

I'll update this ticket when the script is finished, and I'll check for scripts that might be deleting the files. Also, I'll double-check the update_counts scripts, as those look wrongly small.
(Assignee)

Comment 3

6 years ago
OK, I found a problem with the update_counts table, and I think the 2 scripts were conflicting with each other. I've set them to run an hour apart, and fixed the update_counts one.

[root@cm-webdev01-master01 addons_mozilla_org]# ls -lh
total 4.5G
-rwx------ 1 root root 4.5G Feb 28 11:52 addons_mozilla_org.2012.02.28.sql.gz
-rw-r--r-- 1 root root   49 Feb 26 02:10 update_counts.2012.02.26.sql.gz
-rw-r--r-- 1 root root   49 Feb 27 02:10 update_counts.2012.02.27.sql.gz
-rw-r--r-- 1 root root 3.5M Feb 28 12:09 update_counts.2012.02.28.sql.gz

Is that more like what you are expecting?
(Reporter)

Comment 4

6 years ago
that looks good
(Assignee)

Comment 5

6 years ago
The update_counts exports are working again, but for some reason the full backups of addons are not being copied properly.

[root@cm-webdev01-master01 addons_mozilla_org]# ls -l
total 4660808
-rwx------ 1 root root 4760422639 Feb 28 11:52 addons_mozilla_org.2012.02.28.sql.gz
-rw-r--r-- 1 root root         49 Feb 27 02:10 update_counts.2012.02.27.sql.gz
-rw-r--r-- 1 root root    3623454 Feb 28 12:09 update_counts.2012.02.28.sql.gz
-rw-r--r-- 1 root root    3941614 Feb 29 03:00 update_counts.2012.02.29.sql.gz

I'll run the script manually to check and try to figure out why this is still having problems.
(Assignee)

Comment 6

6 years ago
W00t! Finally the timing is right:

[root@cm-webdev01-master01 addons_mozilla_org]# ls -l
total 13985916
-rwx------ 1 root root 4760422639 Feb 28 11:52 addons_mozilla_org.2012.02.28.sql.gz
-rwx------ 1 root root 4764805785 Feb 29 12:41 addons_mozilla_org.2012.02.29.sql.gz
-rwx------ 1 root root 4770812774 Mar  1 04:05 addons_mozilla_org.2012.03.01.sql.gz
-rw-r--r-- 1 root root    3623454 Feb 28 12:09 update_counts.2012.02.28.sql.gz
-rw-r--r-- 1 root root    3941614 Feb 29 03:00 update_counts.2012.02.29.sql.gz
-rw-r--r-- 1 root root    3944196 Mar  1 02:00 update_counts.2012.03.01.sql.gz

Note the 4:05 am copy time of today's backup of addons.

Also note:
[root@cm-webdev01-master01 addons_mozilla_org]# more /etc/cron.d/prune_backups 
0 5 * * * root /usr/bin/find /data/backup-drop/ -type f -mtime +2 -exec rm -f {} \;

The backup from the 28th hasn't been pruned yet because it was only 1.5 days old when the prune script ran. The point is, we don't have to worry about the partition filling up.

Resolving!
Status: ASSIGNED → RESOLVED
Last Resolved: 6 years ago
Resolution: --- → FIXED
(Reporter)

Comment 7

6 years ago
thanks!
(Assignee)

Comment 8

6 years ago
FYI this is happening again, as far as I can tell, due to a netflow issue, so I've opened up bug 779820 for it.
Status: RESOLVED → REOPENED
Depends on: 779820
Resolution: FIXED → ---
(Assignee)

Comment 9

6 years ago
Confirmed that the netflow is now open and backup copies are working again:

[root@webdev1 addons_mozilla_org]# ls -rtlh
total 22G
-rwx------ 1 root root 5.5G Aug 10 12:18 addons_mozilla_org.2012.08.10.sql.gz
-rw-r--r-- 1 root root 3.6M Aug 11 02:00 update_counts.2012.08.11.sql.gz
-rwx------ 1 root root 5.5G Aug 11 04:03 addons_mozilla_org.2012.08.11.sql.gz
-rw-r--r-- 1 root root 3.3M Aug 12 02:00 update_counts.2012.08.12.sql.gz
-rwx------ 1 root root 5.5G Aug 12 04:03 addons_mozilla_org.2012.08.12.sql.gz
-rw-r--r-- 1 root root 3.3M Aug 13 02:00 update_counts.2012.08.13.sql.gz
-rwx------ 1 root root 5.5G Aug 13 04:24 addons_mozilla_org.2012.08.13.sql.gz
Status: REOPENED → RESOLVED
Last Resolved: 6 years ago6 years ago
Resolution: --- → FIXED
Product: mozilla.org → Data & BI Services Team
You need to log in before you can comment on or make changes to this bug.