685746 - [amo] database dumps on cm-webdev01-master01 are stale

Status: RESOLVED FIXED

(Infrastructure & Operations Graveyard :: WebOps: Other, task)

Description

[Wil Clouser [:clouserw]]

cm-webdev01-master01:/data/backup-drop/mrdb04/mysql/addons_remora

We're still getting daily dumps there which is why we're just now noticing this, but judging by file sizes the dumps aren't changing, just doing the same thing over and over.  Last useful dump was from Aug 24th.  We should be getting dumps from the db in phoenix.

Comment 1

[Wil Clouser [:clouserw]]

This is causing us much pain in bug 685669

Comment 2

[Jake Maul [:jakem]]

As you may have guessed, this is broken due to the PHX1 migrations.

The dump that gets placed there is created on tm-backup02, which is a slave of tm-amo-master02. This is one of the original DB masters in SJC. Of course, the data there is no longer being updated, so the dumps don't show any new data.

Dumps on tm-backup02 are all rsync'd to dracula (this is not AMO-specific... all backup dumps do this). Dracula, has a number of cron jobs to copy dumps out to wherever they're needed. This is how the dumps ultimately end up on cm-webdev01-master01.

There is a project underway to set up a DB backup server in PHX1. Once that's done the dumps would originate from there, instead of tm-backup02... this is the longer-term fix, and should be done in the next week or two.

In the interim, we can try to do this manually... that is, make a dump of the data from PHX1 off of a slave, and put that on cm-webdev01-master01 somewhere (and probably disable the cron job that automatically updates the database with the most current copy from tm-backup02, which would be counter-productive at that time).

Whether or not this would be useful to you is hard to judge... the dumps would be similar to the ones discussed in the bug linked in comment 1, which I understand has already been provided to you.

Comment 3

[Corey Shields [:cshields]]

(In reply to Jake Maul [:jakem] from comment #2)
> There is a project underway to set up a DB backup server in PHX1. Once
> that's done the dumps would originate from there, instead of tm-backup02...
> this is the longer-term fix, and should be done in the next week or two.

whoa wait, this should already be done and backing up AMO dumps...   Dumitru? Phong?

Comment 4

[Dumitru Gherman [:dumitru]]

(In reply to Corey Shields [:cshields] from comment #3)
> whoa wait, this should already be done and backing up AMO dumps...  
> Dumitru? Phong?

The backup part of the data is already done. See Bug 682284. Not sure about the status of the sql dumps themselves, mpressman can shed some light here.

Comment 5

[Corey Shields [:cshields]]

We have a more-recent-than-8/24 dump in there right now but I believe the path of getting automatic updates is still broken.

Brian can you work with dumitru on getting the dumps from their backup source into cm-webdev01-master01 in sjc1?  (the dumps are from prod in phx1)

Comment 6

[Brian Hourigan [:digi]]

Awaiting Dumitru's return from PTO to work with him on this.

Comment 7

[Brian Hourigan [:digi]]

Bug 691590 created to allow network flow from db-backup1 to cm-webdev01-master01. An initial dump from today is being loaded into cm-webdev01-master01.remora and the script to pull it from tm-backup02 has been temporarily disabled.

Comment 8

[Brian Hourigan [:digi]]

Data will regularly be dumped to cm-webdev01-master01 from db-backup1 into /data/backup-drop/addons/mysql/addons_mozilla_org nightly at 2AM

Comment 9

[Wil Clouser [:clouserw]]

There's a file from yesterday at 2130 that is empty.  Nothing from 2am.

Comment 10

[Brian Hourigan [:digi]]

Sorry about that, there was a bug in the script. There is a 3.8GB dump file present now.

Comment 11

[Wil Clouser [:clouserw]]

There is a dump from yesterday (thanks!) but nothing from 2am today

Comment 12

[Brian Hourigan [:digi]]

Ack! I'm really sorry about the trouble, Wil. I've made a change and I'm creating a new dump for you now. Let's see if it works tonight at 2AM as intended.

Comment 13

[Brian Hourigan [:digi]]

It looks like it ran last night, please confirm.

Comment 14

[Wil Clouser [:clouserw]]

Looks good to me.  Thanks!